# daftyon/easier-batch architecture

> Architecture diagram of the GitHub repository daftyon/easier-batch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daftyon/easier-batch
- Repository: https://github.com/daftyon/easier-batch

## Overview

##### Project Type & Purpose
- It provides a Reader-Processor-Writer pipeline, transaction management, retries/skips, metadata persistence (MySQL), monitoring, and pluggable extensions.

##### File Structure Highlights
- core/
- item_reader.py, item_processor.py, item_writer.py – abstract base classes and core interfaces
- step.py, job.py, job_launcher.py – orchestration of batch steps and jobs
- metadata_manager.py – manages the JDBC/ORM connection to the metadata tables
- setup.py / easier_batch.egg-info / dist/ – packaging and distribution artifacts
- test_*.py – automated tests for core behavior and database integration
- batch.py – a sample or entry-point script

##### Key Architectural Patterns
- Layered Architecture (Presentation / Business / Infrastructure / Core Engine / Components / Data)
- Reader-Processor-Writer chain inside each Step
- Job → multiple Steps → each Step runs Read → Process → Write with transaction boundaries
- Pluggable implementations (FileReader, DatabaseReader, APIReader, KafkaReader, etc.)
- Cross-cutting concerns (retry/skip, transaction management, monitoring/logging)

##### Main Components & Relationships
A. Presentation Layer
- CLI (batch.py)
- (Optional) REST API or Web Dashboard extensions
↓ invokes
B. Business Layer
- JobLauncher – loads configuration, validates, and kicks off a Job
↓ orchestrates
C. Core Engine
1) Job Execution Engine
– Manages one or more Job instances
– Orchestrates parallel/sequential Step execution
2) Step Execution Engine
– For each Step: opens transaction, calls Chunk Manager, commits/rolls back
↓ within each Step
D. Component Layer
- ItemReader – reads data in chunks
- ItemProcessor – transforms, filters, validates items
- ItemWriter – persists output items
These three form a pipeline:
ItemReader → ItemProcessor → ItemWriter
↑ error handling / retry / skip plugged in at each boundary
E. Infrastructure Layer
- MetadataManager – persists job/step metadata in MySQL tables
- Transaction Manager – JDBC connection pooling, commit/rollback logic
- Retry & Skip Policies – configured via BatchConfiguration
- Monitoring & Logging – collects metrics, health checks, and audit trails
F. Data Layer
- MySQL metadata tables (job_metadata, step_metadata, execution_context)
- (Optional) external data sources: file system, Kafka, REST APIs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers
    subgraph "Presentation Layer"
        CLI["CLI Entry Point\nbatch.py"]:::presentation
    end

    subgraph "Business Layer"
        JobLauncher["JobLauncher"]:::business
        Job["Job"]:::business
        Step["Step"]:::business
    end

    subgraph "Core Engine"
        JobExec["Job Execution Engine"]:::coreengine
        StepExec["Step Execution Engine"]:::coreengine
    end

    subgraph "Component Layer"
        Reader["ItemReader\n(abstract)"]:::component
        Processor["ItemProcessor\n(abstract)"]:::component
        Writer["ItemWriter\n(abstract)"]:::component
    end

    subgraph "Infrastructure Layer"
        MetadataMgr["MetadataManager"]:::infra
        TxMgr["Transaction Manager"]:::infra
        Retry["Retry & Skip Policies"]:::infra
        Monitoring["Monitoring & Logging"]:::infra
    end

    subgraph "Data Layer"
        MySQL[(MySQL\njob_metadata, step_metadata)]:::data
        ExternalData["External Data Sources\n(your_data_file.txt)"]:::external
    end

    subgraph "Packaging & Tests"
        Setup["setup.py"]:::infra
        Dist["dist/"]:::infra
        TestCore["test_batch.py"]:::infra
        TestDB["test_batch_db.py"]:::infra
    end

    %% Main Flow
    CLI -->|"invokes"| JobLauncher
    JobLauncher -->|"launch()"| Job
    Job -->|"defines"| Step
    Job --> JobExec
    JobExec -->|"orchestrates"| StepExec
    StepExec -->|"opens txn"| TxMgr
    StepExec --> Reader
    Reader --> Processor
    Processor --> Writer
    Writer --> StepExec
    StepExec -->|"records status"| MetadataMgr

    %% Persistence
    MetadataMgr -->|"persists via mysql-connector-python"| MySQL

    %% External Sources
    ExternalData --> Reader

    %% Cross-cutting
    Reader ---|dashed retry/skip| Reader
    Processor ---|dashed retry/skip| Processor
    Writer ---|dashed retry/skip| Writer
    Monitoring -.->|collects metrics| StepExec
    Monitoring -.->|collects metrics| JobExec

    %% Packaging & Tests Relationships
    Setup --> Dist
    Setup --> CLI
    Setup --> JobLauncher
    TestCore --> CLI
    TestCore --> JobLauncher
    TestDB --> MetadataMgr

    %% Click Events
    click CLI "https://github.com/daftyon/easier-batch/blob/main/batch.py"
    click JobLauncher "https://github.com/daftyon/easier-batch/blob/main/core/job_launcher.py"
    click Job "https://github.com/daftyon/easier-batch/blob/main/core/job.py"
    click Step "https://github.com/daftyon/easier-batch/blob/main/core/step.py"
    click Reader "https://github.com/daftyon/easier-batch/blob/main/core/item_reader.py"
    click Processor "https://github.com/daftyon/easier-batch/blob/main/core/item_processor.py"
    click Writer "https://github.com/daftyon/easier-batch/blob/main/core/item_writer.py"
    click MetadataMgr "https://github.com/daftyon/easier-batch/blob/main/core/metadata_manager.py"
    click Setup "https://github.com/daftyon/easier-batch/blob/main/setup.py"
    click Dist "https://github.com/daftyon/easier-batch/tree/main/dist/"
    click TestCore "https://github.com/daftyon/easier-batch/blob/main/test_batch.py"
    click TestDB "https://github.com/daftyon/easier-batch/blob/main/test_batch_db.py"
    click ExternalData "https://github.com/daftyon/easier-batch/blob/main/your_data_file.txt"

    %% Styles
    classDef presentation fill:#cfefff,stroke:#036,stroke-width:1px;
    classDef business fill:#e0ffe0,stroke:#060,stroke-width:1px;
    classDef coreengine fill:#e8e8ff,stroke:#004,stroke-width:1px;
    classDef component fill:#ffe0ff,stroke:#600,stroke-width:1px;
    classDef infra fill:#fff0b3,stroke:#aa6600,stroke-width:1px;
    classDef data fill:#ffd6d6,stroke:#900,stroke-width:1px;
    classDef external fill:#f2f2f2,stroke:#666,stroke-width:1px;
```
