# younes921722/data-internship-home-assignment architecture

> Architecture diagram of the GitHub repository younes921722/data-internship-home-assignment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/younes921722/data-internship-home-assignment
- Repository: https://github.com/younes921722/data-internship-home-assignment

## Overview

##### Main Components
- Airflow Orchestrator: At the heart of the system, the Apache Airflow DAG defined in “dags/etl.py” manages the workflow.
- ETL Tasks: Separated into distinct tasks under “dags/tasks” with dedicated modules for:
- Extraction (dags/tasks/extract.py)
- Transformation (dags/tasks/transform.py)
- Loading (dags/tasks/load.py)
- Table creation (dags/tasks/create_tables.py)
- Utility Modules: In “dags/utils” you have helper modules (data_cleaner, data_extractor, data_transformer, db_loader, logger, tables_creator) that support the tasks.
- Data Sources and Sinks:
- Source: “source/jobs.csv” is the input file.
- Staging: Three directories (“staging/extracted”, “staging/transformed”, “staging/loaded”) hold data during each ETL phase (extracted text, transformed JSON, and the SQLite database).
- Testing: The “tests” folder contains unit tests for the ETL tasks, ensuring that data cleaning, extraction, transformation, and loading work as intended.

##### Relationships and Interactions
- The Airflow DAG (orchestrator) triggers each of the ETL tasks in sequence.
- Data flows from the “source” (CSV file) to “staging/extracted” (extraction output).
- Next, the extraction output is consumed by the transformation task to produce files in “staging/transformed”.
- Finally, the loader task takes these JSON files and updates the SQLite database in “staging/loaded”.
- The utility functions reside as supporting modules that are called within each task to handle specific logic (e.g., logging, cleaning, or schema transformation).

##### Architectural Patterns / Design Principles
- Modularization: The code has been refactored into separate Python modules according to their domains (tasks and utilities).
- Separation of Concerns: Each ETL step (Extract, Transform, Load) is isolated; this makes it easier to test and maintain.
- Reusability: Utility functions are re-used across tasks, and the project adheres to clean-code guidelines.
- Automation and Scheduling: Using Apache Airflow for orchestrating and scheduling tasks, which supports scalability and monitoring via the web interface.

##### Relevant Technologies
- Apache Airflow – for defining and executing the ETL pipeline.
- Python – the main programming language.
- SQLite – the target database for the loaded data.
- Pytest – used for unit testing of the ETL tasks.
- Any additional libraries indicated in “requirements.txt” that support ETL operations.

##### Elements
- Clear labeled components:
- "Apache Airflow DAG" (orchestrator)
- "Extract Task" (read CSV → write text file)
- "Transform Task" (read extracted text → write JSON file with the target schema)
- "Load Task" (read transformed JSON → update SQLite DB)
- "Utility Functions" (data_cleaner, data_extractor, etc.)
- "Source Data" (CSV file) and "Staging Layers" (extracted, transformed data folders)
- "SQLite Database" (as the final data store)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        A["Apache Airflow DAG"]:::orchestrator
    end

    %% ETL Tasks
    subgraph "ETL Tasks"
        B["Extract Task"]:::task
        C["Transform Task"]:::task
        D["Create Tables Task"]:::task
        E["Load Task"]:::task
    end

    %% Data Storage
    subgraph "Data Storage"
        F["Source Data (CSV File)"]:::data
        G["Staging (Extracted)"]:::data
        H["Staging (Transformed)"]:::data
        I[("SQLite Database")]:::data
    end

    %% Utility and Testing
    U["Utility Modules"]:::utility
    T["Testing Framework"]:::testing

    %% Data Flow Connections
    A -->|"triggers"| B
    B -->|"outputs_raw_data"| G
    F -->|"provides_CSV"| B
    B -->|"passes_data"| C
    C -->|"outputs_JSON"| H
    C -->|"continues_to"| D
    D -->|"prepares_schema"| E
    H -->|"feeds_JSON"| E
    E -->|"loads_data"| I

    %% Utility Modules supporting each ETL Task
    U --- B
    U --- C
    U --- D
    U --- E

    %% Testing Framework linked to key tasks
    T --- B
    T --- C
    T --- D
    T --- E

    %% Styles
    classDef orchestrator fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef task fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef data fill:#FFFACD,stroke:#000,stroke-width:2px;
    classDef utility fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef testing fill:#FFB6C1,stroke:#000,stroke-width:2px;

    %% Click Events
    click A "https://github.com/younes921722/data-internship-home-assignment/blob/main/dags/etl.py"
    click B "https://github.com/younes921722/data-internship-home-assignment/blob/main/dags/tasks/extract.py"
    click C "https://github.com/younes921722/data-internship-home-assignment/blob/main/dags/tasks/transform.py"
    click D "https://github.com/younes921722/data-internship-home-assignment/blob/main/dags/tasks/create_tables.py"
    click E "https://github.com/younes921722/data-internship-home-assignment/blob/main/dags/tasks/load.py"
    click U "https://github.com/younes921722/data-internship-home-assignment/tree/main/dags/utils"
    click F "https://github.com/younes921722/data-internship-home-assignment/blob/main/source/jobs.csv"
    click G "https://github.com/younes921722/data-internship-home-assignment/tree/main/staging/extracted"
    click H "https://github.com/younes921722/data-internship-home-assignment/tree/main/staging/transformed"
    click I "https://github.com/younes921722/data-internship-home-assignment/tree/main/staging/loaded"
    click T "https://github.com/younes921722/data-internship-home-assignment/tree/main/tests"
```
