# carlosaquinoperez/batch-process architecture

> Architecture diagram of the GitHub repository carlosaquinoperez/batch-process, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/carlosaquinoperez/batch-process
- Repository: https://github.com/carlosaquinoperez/batch-process

## Overview

Based on your project’s file tree and the detailed README, this is a data pipeline project that dynamically processes CSV files in micro-batches using an Apache Airflow orchestration framework, stores computed statistics in a PostgreSQL database, and logs outcomes at various stages. The project is containerized (via Docker Compose) and uses scripts to process CSV files (mainly located in scripts/main.py and within the Airflow DAG definition in dags/data_pipeline_dag.py).

##### Main Components
- Airflow Orchestrator: Represent the Airflow components (scheduler, webserver, and worker if applicable). Indicate that these components use Docker containers.
- PostgreSQL Database: A containerized database that stores transactions and computed statistics.
- Data Processing Logic: The main processing script (scripts/main.py) invoked through the Airflow DAG tasks. This logic reads CSV files from the “dags/data” directory and calculates statistics.
- CSV Data Files: The source files (e.g., 2012-1.csv … 2012-5.csv and validation.csv) that are processed.
- Docker Compose: Illustrate that the Docker Compose file is the bootstrap that connects PostgreSQL, Airflow services, and potential dependencies.
- Logging Mechanism: Include a component for logging (logs folder directories for both DAGs and Airflow’s scheduler) to indicate how log data is captured.

##### Relationships and Interactions
- Data Flow: Show how CSV files (from the “dags/data” directory) are ingested by the processing tasks.
- Processing Outcomes: Illustrate that processed results (statistics such as row count, min, max, avg) are then forwarded to the PostgreSQL database.
- Logging: Connect the processing tasks with the Logs component, showing how outcomes are written as log files.

##### Architectural Patterns & Design Principles
- Micro-batching Pattern: Emphasize that CSV files are processed in batches with a dedicated task for each file, including a specialized task for validation.
- Modular Task Orchestration: Note that the Airflow DAG is segmented into multiple tasks (e.g., process_csv_files, process_validation_csv, log_final_statistics) such that each task is atomic and carries its own logs.
- Containerized Deployment: Highlight the use of Docker Compose for managing dependencies and production environments.
- Data Flow Isolation: Indicate that data is read from a volume-mounted “dags/data” directory, processed by Python scripts, and persisted in PostgreSQL.

##### Relevant Technologies to Note
- Apache Airflow (orchestration)
- PostgreSQL (database engine)
- Python (processing scripts with pandas, psycopg2, and sqlalchemy)
- Docker & Docker Compose (containerization and service orchestration)
- CSV file processing library (pandas)

##### Final Advice
- Be as detailed as possible to capture every significant component (even subcomponents like Airflow’s user interface, scheduler, and the Docker platform).
- Maintain clarity by grouping related items (for instance, all Airflow components within one container boundary).

## Diagram (Mermaid)

```mermaid
flowchart TD
    dcompose["Docker Compose"]:::docker
    csv["CSV Data Files"]:::external
    logs["Log Storage"]:::log

    subgraph "Airflow Container"
        A_sched["Airflow Scheduler"]:::airflow
        A_web["Airflow Webserver"]:::airflow
        DAG["DAG - data_pipeline_dag"]:::airflow
        processing["Data Processing Script (main.py)"]:::python
    end

    subgraph "PostgreSQL Container"
        db["PostgreSQL Database"]:::db
    end

    dcompose -->|"bootstrap"| A_sched
    dcompose -->|"bootstrap"| db

    csv -->|"data_ingestion"| DAG
    A_sched -->|"orchestrates"| DAG
    DAG -->|"executes"| processing
    processing -->|"statistic_reporting"| db
    processing -->|"logs_updates"| logs
    A_web -->|"ui_monitoring"| logs

    click DAG "https://github.com/carlosaquinoperez/batch-process/blob/main/dags/data_pipeline_dag.py"
    click A_sched "https://github.com/carlosaquinoperez/batch-process/tree/main/logs/scheduler"
    click db "https://github.com/carlosaquinoperez/batch-process/blob/main/docker-compose.yml"
    click processing "https://github.com/carlosaquinoperez/batch-process/blob/main/scripts/main.py"
    click csv "https://github.com/carlosaquinoperez/batch-process/tree/main/data/"
    click dcompose "https://github.com/carlosaquinoperez/batch-process/blob/main/docker-compose.yml"
    click logs "https://github.com/carlosaquinoperez/batch-process/tree/main/logs/"

    classDef docker fill:#f9e79f,stroke:#b9770e,stroke-width:2px;
    classDef external fill:#aed6f1,stroke:#5dade2,stroke-width:2px;
    classDef airflow fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef python fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef db fill:#f5b7b1,stroke:#c0392b,stroke-width:2px;
    classDef log fill:#f6e3b4,stroke:#d4ac0d,stroke-width:2px;
```
