How carlosaquinoperez/batch-process works
Based on your project’s file tree and the detailed README, this is a data pipeline project that dynamically processes CSV files in micro-batches using an Apache Airflow orchestration framework, stores computed statistics in a PostgreSQL database, and logs outcomes at various stages. The project is containerized (via Docker Compose) and uses scripts to process CSV files (mainly located in scripts/main.py and within the Airflow DAG definition in dags/data_pipeline_dag.py).
Main Components
• Airflow Orchestrator: Represent the Airflow components (scheduler, webserver, and worker if applicable). Indicate that these components use Docker containers.
• PostgreSQL Database: A containerized database that stores transactions and computed statistics.
• Data Processing Logic: The main processing script (scripts/main.py) invoked through the Airflow DAG tasks. This logic reads CSV files from the “dags/data” directory and calculates statistics.
• CSV Data Files: The source files (e.g., 2012-1.csv … 2012-5.csv and validation.csv) that are processed.
• Docker Compose: Illustrate that the Docker Compose file is the bootstrap that connects PostgreSQL, Airflow services, and potential dependencies.
• Logging Mechanism: Include a component for logging (logs folder directories for both DAGs and Airflow’s scheduler) to indicate how log data is captured.
Relationships and Interactions
• Data Flow: Show how CSV files (from the “dags/data” directory) are ingested by the processing tasks.
• Processing Outcomes: Illustrate that processed results (statistics such as row count, min, max, avg) are then forwarded to the PostgreSQL database.
• Logging: Connect the processing tasks with the Logs component, showing how outcomes are written as log files.
Architectural Patterns & Design Principles
• Micro-batching Pattern: Emphasize that CSV files are processed in batches with a dedicated task for each file, including a specialized task for validation.
• Modular Task Orchestration: Note that the Airflow DAG is segmented into multiple tasks (e.g., process_csv_files, process_validation_csv, log_final_statistics) such that each task is atomic and carries its own logs.
• Containerized Deployment: Highlight the use of Docker Compose for managing dependencies and production environments.
• Data Flow Isolation: Indicate that data is read from a volume-mounted “dags/data” directory, processed by Python scripts, and persisted in PostgreSQL.
Relevant Technologies to Note
• Apache Airflow (orchestration)
• PostgreSQL (database engine)
• Python (processing scripts with pandas, psycopg2, and sqlalchemy)
• Docker & Docker Compose (containerization and service orchestration)
• CSV file processing library (pandas)
Final Advice
• Be as detailed as possible to capture every significant component (even subcomponents like Airflow’s user interface, scheduler, and the Docker platform).
• Maintain clarity by grouping related items (for instance, all Airflow components within one container boundary).