# harsh199323/marketing-attribution-etl-framework--maef architecture

> Architecture diagram of the GitHub repository harsh199323/marketing-attribution-etl-framework--maef, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harsh199323/marketing-attribution-etl-framework--maef
- Repository: https://github.com/harsh199323/marketing-attribution-etl-framework--maef

## Overview

##### Project type and purpose
- This is an Apache Airflow–driven ETL pipeline in Python.
- It automates extraction of raw data, transformation, loading into a database, API integration, and reporting as CSV.

##### File structure
- Top-level:
– main.py (entrypoint for triggering run outside Airflow)
– webserver_config.py (Airflow UI config)
– requirements.txt, README.md
- dirs:
– dags/ihc_api_dag.py → defines the DAG and task dependencies
– src/etl (extract.py, transform.py, load.py, reporting.py) → ETL stages
– src/api (ihc_api.py, api_utils.py) → external API client
– src/database (schema.py, db_utils.py) → DB schema & utilities
– src/utils → date handling
- data/ holds raw SQL, outputs, formatted JSON/CSV.

##### README for additional insights
- Tech stack: Python, Apache Airflow, SQLite (or other RDBMS), external HTTP API.

##### Detail without overcomplexity
- Essential components and their interactions.
- Do not break modules into sub-functions—group at the task level.
- Flow mirrors the DAG’s order exactly.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Scheduler(["Airflow Scheduler"]):::orchestration
        WebUI(["Airflow Webserver UI"]):::orchestration
        MetaDB(["Airflow Metadata DB"]):::persistence
        WebConfig(["Webserver Config"]):::orchestration
        MainEntry(["Main Entrypoint"]):::orchestration
        DAGDef(["DAG Definition"]):::orchestration
    end

    %% ETL Modules
    subgraph "ETL Modules"
        Extractor(["Extractor Module"]):::processing
        Transformer(["Transformer Module"]):::processing
        Loader(["Loader Module"]):::processing
        Reporter(["Reporting Module"]):::processing
    end

    %% API Client Layer
    subgraph "External Service Integration"
        APIClient(["IHCApiClient"]):::external
        APIUtils(["API Utility Functions"]):::processing
    end

    %% Database Layer
    subgraph "Database Layer"
        SchemaInit(["Database Schema Init"]):::persistence
        DBUtils(["Database Utility Functions"]):::persistence
        RDBMS(["Relational DB (SQLite/PostgreSQL)"]):::persistence
    end

    %% Utilities & Data Artifacts
    subgraph "Utilities & Artifacts"
        SQLScript(["SQL Schema Script"]):::artifact
        DateUtils(["Date Handling Utilities"]):::processing
        Output(["Output Artifacts (CSV/JSON)"]):::artifact
    end

    %% Connections
    MainEntry -->|Triggers outside DAG| DAGDef
    Scheduler -->|Schedules| DAGDef
    WebConfig -->|Configures UI| WebUI
    WebUI -->|Displays| DAGDef
    DAGDef -->|Setup DB| SchemaInit
    SchemaInit -->|Applies schema| RDBMS
    DAGDef -->|Extract raw data| Extractor
    Extractor -->|Reads SQL| SQLScript
    Extractor -->|Provides raw data| Transformer
    DAGDef -->|Transform data| Transformer
    Transformer -->|Cleans data| Loader
    DAGDef -->|Load data| Loader
    Loader -->|Writes to DB| DBUtils
    DBUtils -->|Executes writes| RDBMS
    DAGDef -->|Call external API| APIClient
    APIClient -->|Uses utils| APIUtils
    APIClient -->|Receives response| DBUtils
    APIUtils -->|Helper functions| APIClient
    DAGDef -->|Generate report| Reporter
    Reporter -->|Queries DB| DBUtils
    Reporter -->|Formats output| Output
    Reporter -->|Uses date utils| DateUtils

    %% Click Events
    click DAGDef "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/dags/ihc_api_dag.py"
    click WebConfig "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/webserver_config.py"
    click MainEntry "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/main.py"
    click Extractor "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/etl/extract.py"
    click Transformer "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/etl/transform.py"
    click Loader "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/etl/load.py"
    click Reporter "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/etl/reporting.py"
    click APIClient "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/api/ihc_api.py"
    click APIUtils "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/api/api_utils.py"
    click SchemaInit "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/database/schema.py"
    click DBUtils "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/database/db_utils.py"
    click SQLScript "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/data/sql/challenge_db_create.sql"
    click DateUtils "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/blob/main/src/utils/date_utils.py"
    click Output "https://github.com/harsh199323/marketing-attribution-etl-framework--maef/tree/main/data/output/"

    %% Styles
    classDef orchestration fill:#D6E9FF,stroke:#4A90E2,stroke-width:2px;
    classDef processing fill:#E2F7E2,stroke:#34A853,stroke-width:2px;
    classDef persistence fill:#FFE5CC,stroke:#FB8C00,stroke-width:2px;
    classDef external fill:#FDECEA,stroke:#EA4335,stroke-width:2px;
    classDef artifact fill:#F0F0F0,stroke:#9E9E9E,stroke-width:2px;
```
