# sygnowski/doer architecture

> Architecture diagram of the GitHub repository sygnowski/doer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sygnowski/doer
- Repository: https://github.com/sygnowski/doer

## Overview

Based on our review of both the file tree and README, “Doer” is a command-line tool written in Java (using Gradle for build management) that provides a modular set of commands for managing pipelines, Kafka ingestion/dumping, RocksDB operations, Helix toolkit features, and gRPC health checks, among others. The tool uses YAML-based manifest files to configure commands, meaning that its design centers on parsing user inputs and delegating to various domain-specific modules.

##### Project Components & Overall Architecture
- Entrance & Command Dispatcher: Represent the main entry point (e.g., Doer.java) that parses command-line arguments (flags such as -v or command names like “kfeed”, “kdump”, etc.) and dispatches to the corresponding command implementations.
- Domain/Service Layers: Show logical groupings for:
- Pipeline and Record Processing (found under “src/main/java/io/github/s7i/doer/pipeline” and “src/main/java/io/github/s7i/doer/flow”)
- Kafka Interaction (modules under “src/main/java/io/github/s7i/doer/domain/kafka”, including ingest and output handling)
- Manifest processing (under “src/main/java/io/github/s7i/doer/manifest” and its subpackages for dump and ingest)
- RocksDB integration (located in the "rocksdb" module/package)
- Helix integration (in the “helix” package and related classes)
- gRPC and proto processing (modules under “src/main/java/io/github/s7i/doer/domain/grpc” and “proto-doer”)
- Configuration & YAML Manifest Parser: Include a component that handles configuration loading and YAML manifest interpretation, feeding into the corresponding command processing.

##### Relationships & Data Flow
- Data flow begins at the Entry point (main method) which reads command-line arguments, then invokes the Command Dispatcher.
- The Dispatcher routes the request to the appropriate Command Handler. For instance, a “kdump” command would invoke classes in the manifest/dump or kafka/dump packages.
- These command handlers interact with domain layers (e.g., Kafka producers/consumers, proto decoders, or RocksDB wrappers) to process data according to the provided manifest configurations.
- Configuration and environmental data (including YAML manifest files) are parsed by the configuration components and then injected (or passed) into the specific modules.

##### Architectural Patterns & Design Principles
- Modularity: The tool is clearly structured around interchangeable commands and domain-specific modules. Represent these modules as separate, loosely coupled components.
- Command Pattern: The separation of command implementations (each command residing in its own package) is an example of the Command pattern. Indicate a common interface for these commands.
- Configuration-as-Code: The use of manifest files (YAML) suggests a design that decouples configuration from code. Show this as an external component feeding into the system.

##### Relevant Technologies & Frameworks
- Java/Gradle: Represent the build and runtime environment.
- YAML for configuration: Indicate that manifest files in YAML format drive workflows.
- Kafka, RocksDB, PROTO, Helix, and gRPC: Label connections to these external systems or protocols.
- Docker and Jenkins: Although not core to the runtime architecture, include a note that CI/CD (via Jenkins and GitHub workflows) and containerization (Dockerfiles) exist to support deployments.

##### Instructing the Principal Software Engineer
- Use clear labels for each component to indicate responsibilities and interactions.
- Aim to detail every major package group (commands, domain logic, manifest parsing, and integrations) so that any changes or troubleshooting in one area are quickly obvious.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Layer"
        U("User Input"):::cli
        Entry("Doer (Entry Point)"):::entry
    end

    %% Command Processing Layer
    subgraph "Command Processing Layer"
        Dispatcher("Command Dispatcher"):::dispatcher
        Config("YAML Manifest Parser & Config"):::config
        subgraph "Command Handlers"
            CommandInterface("Generic Command Interface"):::command
            GRPC("gRPC Health Command"):::command
            HelixCmd("Helix Command"):::command
            KafkaIngest("Kafka Ingestion Command"):::command
            KafkaDump("Kafka Dump Command"):::command
            RocksDBCmd("RocksDB Command"):::command
            ProtoProc("Proto Processing Command"):::command
            Sink("Sink/Misc Command"):::command
        end
    end

    %% Domain & Service Layer
    subgraph "Domain & Service Layer"
        subgraph "Pipeline/Flow Processing"
            Pipeline("Pipeline Modules"):::service
            Flow("Flow Modules"):::service
        end
        subgraph "Kafka Integration"
            KafkaDomain("Kafka Domain Logic"):::service
            KafkaDumpDomain("Kafka Dump Submodule"):::service
            KafkaIngestDomain("Kafka Ingest Submodule"):::service
            KafkaOutput("Kafka Output Handling"):::service
        end
        subgraph "Manifest Processing"
            ManifestCore("Manifest Core"):::service
            ManifestDump("Manifest Dump Handling"):::service
            ManifestIngest("Manifest Ingest Handling"):::service
            ProtoManifest("Proto Manifest"):::service
        end
        RocksDBDomain("RocksDB Integration"):::service
        HelixDomain("Helix Integration"):::service
        subgraph "gRPC & Proto Processing"
            GRPCServer("gRPC Server"):::service
            ProtoDef("Proto Definitions & Processing"):::service
        end
    end

    %% External Systems
    subgraph "External Systems"
        KafkaExt(("Kafka Cluster")):::external
        RocksDBExt(("RocksDB Storage")):::external
        HelixExt(("Helix Controller")):::external
        GRPCExt(("gRPC Endpoint")):::external
    end

    %% Connections: CLI to Command Layer
    U -->|"CLI_Command"| Entry
    Entry -->|"parses"| Dispatcher

    %% Dispatcher to Command Handlers & Config
    Dispatcher -->|"loads"| Config
    Dispatcher -->|"routes"| CommandInterface
    Dispatcher -->|"routes"| GRPC
    Dispatcher -->|"routes"| HelixCmd
    Dispatcher -->|"routes"| KafkaIngest
    Dispatcher -->|"routes"| KafkaDump
    Dispatcher -->|"routes"| RocksDBCmd
    Dispatcher -->|"routes"| ProtoProc
    Dispatcher -->|"routes"| Sink

    %% Config to Manifest Core
    Config -->|"configures"| ManifestCore

    %% Command Handlers to Domain Components
    CommandInterface -->|"triggers"| Pipeline
    CommandInterface -->|"triggers"| Flow
    KafkaIngest -->|"processes"| KafkaIngestDomain
    KafkaDump -->|"processes"| KafkaDumpDomain
    GRPC -->|"healthchecks"| GRPCServer
    HelixCmd -->|"controls"| HelixDomain
    RocksDBCmd -->|"db_operations"| RocksDBDomain
    ProtoProc -->|"processes"| ProtoManifest

    %% Domain Relationships
    KafkaDomain --> KafkaIngestDomain
    KafkaDomain --> KafkaDumpDomain
    KafkaDomain --> KafkaOutput
    ManifestCore --> ManifestDump
    ManifestCore --> ManifestIngest

    %% Domain to External Systems
    KafkaIngestDomain -->|"writes"| KafkaExt
    KafkaDumpDomain -->|"reads"| KafkaExt
    KafkaOutput --> KafkaExt
    RocksDBDomain -->|"stores"| RocksDBExt
    HelixDomain -->|"orchestrates"| HelixExt
    GRPCServer -->|"exposes"| GRPCExt

    %% Style Declarations
    classDef cli fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef entry fill:#A9DFBF,stroke:#145A32,stroke-width:2px;
    classDef dispatcher fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef command fill:#FAD7A0,stroke:#B03A2E,stroke-width:2px;
    classDef config fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef service fill:#F5CBA7,stroke:#873600,stroke-width:2px;
    classDef external fill:#E8DAEF,stroke:#6C3483,stroke-width:2px;

    %% Click Events
    click Entry "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/Doer.java"
    click CommandInterface "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/Command.java"
    click GRPC "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/GrpcHealth.java"
    click HelixCmd "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/Helix.java"
    click KafkaIngest "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/Ingest.java"
    click KafkaDump "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/dump/KafkaDump.java"
    click RocksDBCmd "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/Rocks.java"
    click ProtoProc "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/ProtoProcessor.java"
    click Sink "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/SinkCommand.java"
    click Pipeline "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/pipeline"
    click Flow "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/flow"
    click KafkaDomain "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/domain/kafka"
    click KafkaDumpDomain "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/domain/kafka/dump"
    click KafkaIngestDomain "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/domain/kafka/ingest"
    click KafkaOutput "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/domain/kafka/output"
    click ManifestCore "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/manifest"
    click ManifestDump "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/manifest/dump"
    click ManifestIngest "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/manifest/ingest"
    click ProtoManifest "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/manifest/proto"
    click RocksDBDomain "https://github.com/sygnowski/doer/tree/main/rocksdb/src/main/java/io/github/s7i/doer/domain/rocksdb"
    click HelixDomain "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/domain/helix"
    click GRPCServer "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/domain/grpc/GrpcServer.java"
    click ProtoDef "https://github.com/sygnowski/doer/tree/main/proto-doer/src/main/proto"
    click Config "https://github.com/sygnowski/doer/blob/main/src/main/java/io/github/s7i/doer/command/YamlParser.java"
    click Config "https://github.com/sygnowski/doer/tree/main/src/main/java/io/github/s7i/doer/config"
```
