# zain-ul-abdin45/analytics-engineer-project architecture

> Architecture diagram of the GitHub repository zain-ul-abdin45/analytics-engineer-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zain-ul-abdin45/analytics-engineer-project
- Repository: https://github.com/zain-ul-abdin45/analytics-engineer-project

## Overview

##### Project Type and Purpose
- It is not a traditional web app but an end-to-end data pipeline with four layers: Ingestion, Processing, Storage, and Visualization.

##### File Structure Analysis
- Top level:
- data/ ─ raw data design docs and (presumably) raw data files (Parquet, JSON).
- docs/ ─ design docs and PDF of final dashboards.
- src/ ─ Python scripts and Jupyter notebook for ETL and EDA.
- README.md, requirements.txt.
- ETL scripts in src:
- ETL_client_part_collection_data_processing_scripts.py
- ETL_data_processing_sonar_results.py
- EDA notebook in src:
- EDA_Markt_Pilot.ipynb

##### Architecture Overview
The pipeline comprises four main components:
A. Raw Data Sources
B. ETL & EDA Processing
C. Relational Storage
D. Dashboarding & Visualization

##### Components and Interactions
a. Raw Data Sources (Layer 1)
- Files in data/: Parquet files with nested JSON fields (e.g., sonar_results).
- Technology: DuckDB for on-the-fly querying, minimal memory.

b. ETL & EDA Processing (Layer 2)
- Two Python ETL scripts that:
- Use pandas + duckdb to extract and normalize nested JSON.
- Clean, standardize, and transform data.
- Load into PostgreSQL via SQLAlchemy/psycopg2.
- One Jupyter notebook (EDA_Markt_Pilot.ipynb) that:
- Reads cleaned data (either via DuckDB or PostgreSQL).
- Performs exploratory analyses with pandas, seaborn, matplotlib.
- Execution: Manual or scheduled via cron/CI (not explicitly in repo).

c. Relational Storage (Layer 3)
- PostgreSQL instance running locally.
- Schema mirrors cleaned data: tables for parts, shops, prices, delivery_times, etc.
- Indexed for query performance.

d. Dashboarding & Visualization (Layer 4)
- Apache Superset connected to PostgreSQL.
- Dashboards: country distributions, price trends, delivery-time histograms, completeness heatmaps.
- Final dashboards exported as PDF (in docs/).

##### Architectural Patterns & Principles
- Layered architecture (Ingestion → Processing → Storage → Presentation).
- Separation of concerns:
- ETL scripts are distinct from EDA notebooks.
- Data storage is decoupled from visualization.
- Use of lightweight in-process querying (DuckDB) for speed, then durable storage in PostgreSQL.
- Modular: each script/notebook handles a single responsibility.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: Data Sources
    subgraph "Layer 1: Data Sources"
        direction TB
        RawData["Parquet, JSON raw data (data/)"]:::dataSource
    end

    %% Layer 2: Processing
    subgraph "Layer 2: ETL & EDA Processing"
        direction TB
        ETL_A["ETL Script A: Client Part Collection"]:::processing
        ETL_B["ETL Script B: Sonar Results Processing"]:::processing
        Notebook["Jupyter EDA Notebook"]:::processing
    end

    %% Layer 3: Storage
    subgraph "Layer 3: Relational Storage"
        direction TB
        PostgresDB["PostgreSQL Database"]:::storage
    end

    %% Layer 4: Visualization
    subgraph "Layer 4: Dashboarding & Visualization"
        direction TB
        Superset["Apache Superset"]:::vis
    end

    %% Documentation Supporting Layers 1 & 2
    subgraph "Supporting Documentation"
        direction TB
        DataDesignDoc["data_design_document.md"]:::docs
        TechReqDoc["technical_requirement_and_implementaion.md"]:::docs
        README["README.md"]:::docs
        Requirements["requirements.txt"]:::docs
    end

    %% Final Dashboards
    FinalPDF["Dashboard PDF"]:::docs

    %% Connections
    RawData -->|extract| ETL_A
    RawData -->|extract| ETL_B
    ETL_A -->|load via SQLAlchemy/psycopg2| PostgresDB
    ETL_B -->|load via SQLAlchemy/psycopg2| PostgresDB
    PostgresDB -->|read| Notebook
    Notebook -->|schema exploration| PostgresDB
    PostgresDB -->|query| Superset
    Superset -->|export PDF| FinalPDF

    %% Click Events
    click RawData "https://github.com/zain-ul-abdin45/analytics-engineer-project/tree/main/data/"
    click ETL_A "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/src/ETL_client_part_collection_data_processing_scripts.py"
    click ETL_B "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/src/ETL_data_processing_sonar_results.py"
    click Notebook "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/src/EDA_Markt_Pilot.ipynb"
    click DataDesignDoc "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/data/data_design_document.md"
    click TechReqDoc "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/docs/technical_requirement_and_implementaion.md"
    click README "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/README.md"
    click Requirements "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/requirements.txt"
    click FinalPDF "https://github.com/zain-ul-abdin45/analytics-engineer-project/blob/main/docs/markt-pilot-supplier-price-point-analysis-platform-2025-04-26.pdf"

    %% Styles
    classDef dataSource fill:#ccf,stroke:#333,stroke-width:1px
    classDef processing fill:#cfc,stroke:#333,stroke-width:1px
    classDef storage fill:#ffc,stroke:#333,stroke-width:1px
    classDef vis fill:#fcc,stroke:#333,stroke-width:1px
    classDef docs fill:#eee,stroke:#999,stroke-width:1px,stroke-dasharray: 2 2
```
