# john-mwangi/data_pipeline architecture

> Architecture diagram of the GitHub repository john-mwangi/data_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/john-mwangi/data_pipeline
- Repository: https://github.com/john-mwangi/data_pipeline

## Overview

- Data Sources: Represent the origins of your data. Include components like CSV Files, JSON Files, and Remote URLs since these are mentioned in both the file_tree and the README.
- Fetch Data – implemented in the main pipeline (main.py and pipeline.py).
- Process & Clean Data – includes operations like name splitting and date conversions.
- Validate Data – capture usage of Pandera for runtime type checking.
- Calculate Data Quality Metrics – represents the computation of categorical counts, min/max values, and null counts.
- SQLite Database: Display the database layer where processed data and metrics are stored (as seen in the file tree with pipeline.db and references in the README to processed_data and data_quality tables).
- API Layer: Illustrate the REST API architecture built on FastAPI with additional functions:
- API Server (FastAPI implementation in api.py)
- Authentication (Basic Auth configured via config.yaml)
- Rate Limiting and Pagination support, illustrating middleware or dedicated components that manage these aspects.

##### Interactions and data flow between components
- Within the ETL pipeline, chain the processes from fetching → processing → validation → metrics computation.
- Database to the API layer, showing that the FastAPI server retrieves data for client requests.

##### Incorporate architectural patterns and design principles
- Separation of concerns: ETL processing is decoupled from the API serving logic.
- Modularity in the codebase (separate directories for data, src, tests, and output) to reflect maintainability.
- Use consistent naming and clear boundaries (e.g., Data Sources vs. ETL Pipeline vs. API Layer).

##### Specific technologies
- Use of Polars for data processing.
- Call out FastAPI for the API server.
- Explicitly note the use of Pandera in the data validation step.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Data Sources"
        CSV["CSV Files"]:::datasource
        JSON["JSON Files"]:::datasource
        Remote["Remote URLs"]:::datasource
    end

    subgraph "ETL Pipeline"
        Fetch["Fetch Data"]:::etl
        Process["Process & Clean Data (Polars)"]:::etl
        Validate["Validate Data (Pandera)"]:::etl
        Metrics["Calculate Data Quality Metrics"]:::etl
    end

    subgraph "Database"
        DB["SQLite Database"]:::database
        Output["Processed Data Output"]:::database
    end

    subgraph "API Layer"
        API["API Server (FastAPI)"]:::api
        Config["API Configuration & Security"]:::api
    end

    Client["API Client"]:::client

    CSV --> Fetch
    JSON --> Fetch
    Remote --> Fetch

    Fetch --> Process
    Process --> Validate
    Validate --> Metrics

    Metrics --> DB
    Metrics --> Output

    DB --> API
    Output --> API

    Config --> API

    Client --> API

    click CSV "https://github.com/john-mwangi/data_pipeline/tree/main/data_pipeline/data"
    click Fetch "https://github.com/john-mwangi/data_pipeline/blob/main/data_pipeline/main.py"
    click Process "https://github.com/john-mwangi/data_pipeline/blob/main/data_pipeline/src/pipeline.py"
    click API "https://github.com/john-mwangi/data_pipeline/blob/main/data_pipeline/src/api.py"
    click Config "https://github.com/john-mwangi/data_pipeline/blob/main/data_pipeline/src/config.yaml"
    click Output "https://github.com/john-mwangi/data_pipeline/tree/main/data_pipeline/output"

    classDef datasource fill:#FFD700,stroke:#8B8000,stroke-width:2px;
    classDef etl fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef database fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef api fill:#FFB6C1,stroke:#C71585,stroke-width:2px;
    classDef client fill:#D3D3D3,stroke:#696969,stroke-width:2px;
```
