How ferozewaris/data-framework works
This repository is a configurable Gen-AI data pipeline. A user supplies a data path and dashboard request; the entry point generates a YAML configuration, ingests data, cleans and aggregates it, then creates and saves charts. The graph also shows the documented alternate ingestion sources and LLM providers. Edges are grounded in the README or shown calls; external services are modeled separately.
Main entry points: llm_config_generator.py Config generator · main.py Pipeline runner · ingestion.py Data ingestion · utils.py YAML writer
Components 21
Pipeline orchestration
- Pipeline runner
main.py - Data ingestion
ingestion.py
AI configuration
- Config generator
llm_config_generator.py - Schema detection
utils.py - Provider dispatch
llm_config_generator.py - YAML writer
utils.py
Data processing
- Data cleaning
cleaning.py - Data aggregation
aggregation.py
Dashboard output
- Chart generation
dashboard.py - Dashboard saver
dashboard.py
Other components
- User
- Hugging Face
- OpenAI
- Gemini
- Replicate
- Generated config
- CSV file
- SQL database
- MongoDB
- API service
- HTML dashboard
Connections 24
- User → Pipeline runner: provides path and intent
README.md - Pipeline runner → Config generator: requests config
main.py - Config generator → Schema detection: detects schema
llm_config_generator.py - Config generator → Provider dispatch: queries provider
llm_config_generator.py - Provider dispatch → Hugging Face: dispatches to
llm_config_generator.py - Provider dispatch → OpenAI: dispatches to
llm_config_generator.py - Provider dispatch → Gemini: dispatches to
llm_config_generator.py - Provider dispatch → Replicate: dispatches to
llm_config_generator.py - Hugging Face → Config generator: returns generated text
- OpenAI → Config generator: returns generated text
- Gemini → Config generator: returns generated text
- Replicate → Config generator: returns generated text
- Config generator → YAML writer: writes config
llm_config_generator.py - YAML writer → Generated config: writes YAML
utils.py - Pipeline runner → Data ingestion: loads CSV
main.py - Data ingestion → CSV file: reads CSV
ingestion.py - Data ingestion → SQL database: queries database
ingestion.py - Data ingestion → MongoDB: reads collection
ingestion.py - Data ingestion → API service: requests data
ingestion.py - Pipeline runner → Data cleaning: cleans data
main.py - Pipeline runner → Data aggregation: aggregates cleaned data
main.py - Pipeline runner → Chart generation: generates charts
main.py - Pipeline runner → Dashboard saver: saves charts
main.py - Dashboard saver → HTML dashboard: writes HTML
dashboard.py