How sygnowski/doer works
Based on our review of both the file tree and README, “Doer” is a command-line tool written in Java (using Gradle for build management) that provides a modular set of commands for managing pipelines, Kafka ingestion/dumping, RocksDB operations, Helix toolkit features, and gRPC health checks, among others. The tool uses YAML-based manifest files to configure commands, meaning that its design centers on parsing user inputs and delegating to various domain-specific modules.
Project Components & Overall Architecture
• Entrance & Command Dispatcher: Represent the main entry point (e.g., Doer.java) that parses command-line arguments (flags such as -v or command names like “kfeed”, “kdump”, etc.) and dispatches to the corresponding command implementations.
• Domain/Service Layers: Show logical groupings for:
• Pipeline and Record Processing (found under “src/main/java/io/github/s7i/doer/pipeline” and “src/main/java/io/github/s7i/doer/flow”)
• Kafka Interaction (modules under “src/main/java/io/github/s7i/doer/domain/kafka”, including ingest and output handling)
• Manifest processing (under “src/main/java/io/github/s7i/doer/manifest” and its subpackages for dump and ingest)
• RocksDB integration (located in the "rocksdb" module/package)
• Helix integration (in the “helix” package and related classes)
• gRPC and proto processing (modules under “src/main/java/io/github/s7i/doer/domain/grpc” and “proto-doer”)
• Configuration & YAML Manifest Parser: Include a component that handles configuration loading and YAML manifest interpretation, feeding into the corresponding command processing.
Relationships & Data Flow
• Data flow begins at the Entry point (main method) which reads command-line arguments, then invokes the Command Dispatcher.
• The Dispatcher routes the request to the appropriate Command Handler. For instance, a “kdump” command would invoke classes in the manifest/dump or kafka/dump packages.
• These command handlers interact with domain layers (e.g., Kafka producers/consumers, proto decoders, or RocksDB wrappers) to process data according to the provided manifest configurations.
• Configuration and environmental data (including YAML manifest files) are parsed by the configuration components and then injected (or passed) into the specific modules.
Architectural Patterns & Design Principles
• Modularity: The tool is clearly structured around interchangeable commands and domain-specific modules. Represent these modules as separate, loosely coupled components.
• Command Pattern: The separation of command implementations (each command residing in its own package) is an example of the Command pattern. Indicate a common interface for these commands.
• Configuration-as-Code: The use of manifest files (YAML) suggests a design that decouples configuration from code. Show this as an external component feeding into the system.
Relevant Technologies & Frameworks
• Java/Gradle: Represent the build and runtime environment.
• YAML for configuration: Indicate that manifest files in YAML format drive workflows.
• Kafka, RocksDB, PROTO, Helix, and gRPC: Label connections to these external systems or protocols.
• Docker and Jenkins: Although not core to the runtime architecture, include a note that CI/CD (via Jenkins and GitHub workflows) and containerization (Dockerfiles) exist to support deployments.
Instructing the Principal Software Engineer
• Use clear labels for each component to indicate responsibilities and interactions.
• Aim to detail every major package group (commands, domain logic, manifest parsing, and integrations) so that any changes or troubleshooting in one area are quickly obvious.