How younes921722/data-internship-home-assignment works
Main Components
• Airflow Orchestrator: At the heart of the system, the Apache Airflow DAG defined in “dags/etl.py” manages the workflow.
• ETL Tasks: Separated into distinct tasks under “dags/tasks” with dedicated modules for:
• Extraction (dags/tasks/extract.py)
• Transformation (dags/tasks/transform.py)
• Loading (dags/tasks/load.py)
• Table creation (dags/tasks/create_tables.py)
• Utility Modules: In “dags/utils” you have helper modules (data_cleaner, data_extractor, data_transformer, db_loader, logger, tables_creator) that support the tasks.
• Data Sources and Sinks:
• Source: “source/jobs.csv” is the input file.
• Staging: Three directories (“staging/extracted”, “staging/transformed”, “staging/loaded”) hold data during each ETL phase (extracted text, transformed JSON, and the SQLite database).
• Testing: The “tests” folder contains unit tests for the ETL tasks, ensuring that data cleaning, extraction, transformation, and loading work as intended.
Relationships and Interactions
• The Airflow DAG (orchestrator) triggers each of the ETL tasks in sequence.
• Data flows from the “source” (CSV file) to “staging/extracted” (extraction output).
• Next, the extraction output is consumed by the transformation task to produce files in “staging/transformed”.
• Finally, the loader task takes these JSON files and updates the SQLite database in “staging/loaded”.
• The utility functions reside as supporting modules that are called within each task to handle specific logic (e.g., logging, cleaning, or schema transformation).
Architectural Patterns / Design Principles
• Modularization: The code has been refactored into separate Python modules according to their domains (tasks and utilities).
• Separation of Concerns: Each ETL step (Extract, Transform, Load) is isolated; this makes it easier to test and maintain.
• Reusability: Utility functions are re-used across tasks, and the project adheres to clean-code guidelines.
• Automation and Scheduling: Using Apache Airflow for orchestrating and scheduling tasks, which supports scalability and monitoring via the web interface.
Relevant Technologies
• Apache Airflow – for defining and executing the ETL pipeline.
• Python – the main programming language.
• SQLite – the target database for the loaded data.
• Pytest – used for unit testing of the ETL tasks.
• Any additional libraries indicated in “requirements.txt” that support ETL operations.
Elements
• Clear labeled components:
• "Apache Airflow DAG" (orchestrator)
• "Extract Task" (read CSV → write text file)
• "Transform Task" (read extracted text → write JSON file with the target schema)
• "Load Task" (read transformed JSON → update SQLite DB)
• "Utility Functions" (data_cleaner, data_extractor, etc.)
• "Source Data" (CSV file) and "Staging Layers" (extracted, transformed data folders)
• "SQLite Database" (as the final data store)