# ferozewaris/data-framework architecture

> Architecture diagram of the GitHub repository ferozewaris/data-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ferozewaris/data-framework
- Repository: https://github.com/ferozewaris/data-framework

## Overview

This repository is a configurable Gen-AI data pipeline. A user supplies a data path and dashboard request; the entry point generates a YAML configuration, ingests data, cleans and aggregates it, then creates and saves charts. The graph also shows the documented alternate ingestion sources and LLM providers. Edges are grounded in the README or shown calls; external services are modeled separately.

Main entry points: [`llm_config_generator.py`](https://github.com/ferozewaris/data-framework/blob/main/llm_config_generator.py) (Config generator), [`main.py`](https://github.com/ferozewaris/data-framework/blob/main/main.py) (Pipeline runner), [`ingestion.py`](https://github.com/ferozewaris/data-framework/blob/main/ingestion.py) (Data ingestion), [`utils.py`](https://github.com/ferozewaris/data-framework/blob/main/utils.py) (YAML writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_main["Pipeline runner<br/>[main.py]"]
  node_ingest["Data ingestion<br/>[ingestion.py]"]
end

subgraph group_configuration["AI configuration"]
  node_configgen["Config generator"]
  node_schema["Schema detection<br/>[utils.py]"]
  node_dispatch["Provider dispatch"]
  node_yamlwriter["YAML writer<br/>[utils.py]"]
end

subgraph group_transforms["Data processing"]
  node_clean["Data cleaning<br/>[cleaning.py]"]
  node_aggregate["Data aggregation<br/>[aggregation.py]"]
end

subgraph group_presentation["Dashboard output"]
  node_charts["Chart generation<br/>[dashboard.py]"]
  node_save["Dashboard saver<br/>[dashboard.py]"]
end

node_user(("User"))
node_huggingface["Hugging Face"]
node_openai["OpenAI"]
node_gemini["Gemini"]
node_replicate["Replicate"]
node_configfile["Generated config"]
node_csv["CSV file"]
node_sql[("SQL database")]
node_mongo[("MongoDB")]
node_api["API service"]
node_html["HTML dashboard"]

node_user -->|"provides path and intent"| node_main
node_main -->|"requests config"| node_configgen
node_configgen -->|"detects schema"| node_schema
node_configgen -->|"queries provider"| node_dispatch
node_dispatch -->|"dispatches to"| node_huggingface
node_dispatch -->|"dispatches to"| node_openai
node_dispatch -->|"dispatches to"| node_gemini
node_dispatch -->|"dispatches to"| node_replicate
node_huggingface -.->|"returns generated text"| node_configgen
node_openai -.->|"returns generated text"| node_configgen
node_gemini -.->|"returns generated text"| node_configgen
node_replicate -.->|"returns generated text"| node_configgen
node_configgen -->|"writes config"| node_yamlwriter
node_yamlwriter -->|"writes YAML"| node_configfile
node_main -->|"loads CSV"| node_ingest
node_ingest -->|"reads CSV"| node_csv
node_ingest -->|"queries database"| node_sql
node_ingest -->|"reads collection"| node_mongo
node_ingest -->|"requests data"| node_api
node_main -->|"cleans data"| node_clean
node_main -->|"aggregates cleaned data"| node_aggregate
node_main -->|"generates charts"| node_charts
node_main -->|"saves charts"| node_save
node_save -->|"writes HTML"| node_html

click node_main "https://github.com/ferozewaris/data-framework/blob/main/main.py"
click node_configgen "https://github.com/ferozewaris/data-framework/blob/main/llm_config_generator.py"
click node_schema "https://github.com/ferozewaris/data-framework/blob/main/utils.py"
click node_dispatch "https://github.com/ferozewaris/data-framework/blob/main/llm_config_generator.py"
click node_yamlwriter "https://github.com/ferozewaris/data-framework/blob/main/utils.py"
click node_ingest "https://github.com/ferozewaris/data-framework/blob/main/ingestion.py"
click node_clean "https://github.com/ferozewaris/data-framework/blob/main/cleaning.py"
click node_aggregate "https://github.com/ferozewaris/data-framework/blob/main/aggregation.py"
click node_charts "https://github.com/ferozewaris/data-framework/blob/main/dashboard.py"
click node_save "https://github.com/ferozewaris/data-framework/blob/main/dashboard.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_ingest,node_user toneBlue
class node_configgen,node_schema,node_dispatch,node_yamlwriter,node_sql,node_mongo toneAmber
class node_clean,node_aggregate,node_api toneMint
class node_charts,node_save toneRose
class node_huggingface,node_openai,node_gemini,node_replicate,node_configfile,node_csv,node_html toneIndigo
```

## Components

### Pipeline orchestration

- [Pipeline runner](https://github.com/ferozewaris/data-framework/blob/main/main.py): `main.py`
- [Data ingestion](https://github.com/ferozewaris/data-framework/blob/main/ingestion.py): `ingestion.py`

### AI configuration

- [Config generator](https://github.com/ferozewaris/data-framework/blob/main/llm_config_generator.py): `llm_config_generator.py`
- [Schema detection](https://github.com/ferozewaris/data-framework/blob/main/utils.py): `utils.py`
- [Provider dispatch](https://github.com/ferozewaris/data-framework/blob/main/llm_config_generator.py): `llm_config_generator.py`
- [YAML writer](https://github.com/ferozewaris/data-framework/blob/main/utils.py): `utils.py`

### Data processing

- [Data cleaning](https://github.com/ferozewaris/data-framework/blob/main/cleaning.py): `cleaning.py`
- [Data aggregation](https://github.com/ferozewaris/data-framework/blob/main/aggregation.py): `aggregation.py`

### Dashboard output

- [Chart generation](https://github.com/ferozewaris/data-framework/blob/main/dashboard.py): `dashboard.py`
- [Dashboard saver](https://github.com/ferozewaris/data-framework/blob/main/dashboard.py): `dashboard.py`

### Other components

- **User**
- **Hugging Face**
- **OpenAI**
- **Gemini**
- **Replicate**
- **Generated config**
- **CSV file**
- **SQL database**
- **MongoDB**
- **API service**
- **HTML dashboard**

## Connections

- User → Pipeline runner: provides path and intent (evidence: `README.md`)
- Pipeline runner → Config generator: requests config (evidence: `main.py`)
- Config generator → Schema detection: detects schema (evidence: `llm_config_generator.py`)
- Config generator → Provider dispatch: queries provider (evidence: `llm_config_generator.py`)
- Provider dispatch → Hugging Face: dispatches to (evidence: `llm_config_generator.py`)
- Provider dispatch → OpenAI: dispatches to (evidence: `llm_config_generator.py`)
- Provider dispatch → Gemini: dispatches to (evidence: `llm_config_generator.py`)
- Provider dispatch → Replicate: dispatches to (evidence: `llm_config_generator.py`)
- Hugging Face → Config generator: returns generated text
- OpenAI → Config generator: returns generated text
- Gemini → Config generator: returns generated text
- Replicate → Config generator: returns generated text
- Config generator → YAML writer: writes config (evidence: `llm_config_generator.py`)
- YAML writer → Generated config: writes YAML (evidence: `utils.py`)
- Pipeline runner → Data ingestion: loads CSV (evidence: `main.py`)
- Data ingestion → CSV file: reads CSV (evidence: `ingestion.py`)
- Data ingestion → SQL database: queries database (evidence: `ingestion.py`)
- Data ingestion → MongoDB: reads collection (evidence: `ingestion.py`)
- Data ingestion → API service: requests data (evidence: `ingestion.py`)
- Pipeline runner → Data cleaning: cleans data (evidence: `main.py`)
- Pipeline runner → Data aggregation: aggregates cleaned data (evidence: `main.py`)
- Pipeline runner → Chart generation: generates charts (evidence: `main.py`)
- Pipeline runner → Dashboard saver: saves charts (evidence: `main.py`)
- Dashboard saver → HTML dashboard: writes HTML (evidence: `dashboard.py`)
