# ahmednasser5/datapilot-ai architecture

> Architecture diagram of the GitHub repository ahmednasser5/datapilot-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmednasser5/datapilot-ai
- Repository: https://github.com/ahmednasser5/datapilot-ai

## Overview

DataPilot AI is a Streamlit-based, deterministic-first analytics platform: users upload datasets, inspect profiles and quality findings, optionally approve explainable cleaning changes, then analyze, query, visualize, and export results. WorkflowService coordinates analysis and re-runs downstream analysis after a cleaned version is promoted. The README documents the major product capabilities; source excerpts clarify workflow, cleaning, and state ownership, but many engines and pages were not sampled, so their internal call relationships are not asserted.

Main entry points: [`services/cleaning_service.py`](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/cleaning_service.py) (Cleaning service), [`services/workflow_service.py`](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/workflow_service.py) (Workflow service), [`streamlit_app/pages/05_Cleaning_Studio.py`](https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/pages/05_Cleaning_Studio.py) (Cleaning studio), [`services/analytics_service.py`](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/analytics_service.py) (Analytics and AI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User experience"]
  node_ui["Streamlit interface<br/>[app.py]"]
  node_upload_ui["Upload and pages<br/>[02_Upload_Data.py]"]
  node_cleaning_ui["Cleaning studio"]
end

subgraph group_workflow["Application workflow"]
  node_workflow_service["Workflow service"]
  node_dataset_service["Dataset service<br/>[dataset_service.py]"]
  node_cleaning_service["Cleaning service"]
end

subgraph group_data_engines["Data engines"]
  node_ingestion["Format loaders<br/>[loader_registry.py]"]
  node_profiling["Profiling<br/>[profiler.py]"]
  node_quality["Quality assessment<br/>[quality_engine.py]"]
  node_cleaning_plan["Cleaning planner"]
  node_cleaning_preview["Cleaning preview<br/>[preview.py]"]
  node_cleaning_executor["Cleaning executor"]
  node_validation["Cleaning validation"]
  node_sql["SQL analytics<br/>[query_executor.py]"]
  node_duckdb[("DuckDB engine<br/>[duckdb_engine.py]")]
end

subgraph group_intelligence["Analysis and AI"]
  node_analytics["Analytics and AI"]
  node_kpi["Domain KPIs<br/>[kpi_calculator.py]"]
  node_dashboard["Dashboard planning"]
  node_ai["AI reasoning<br/>[analyst_agent.py]"]
end

subgraph group_outputs["Outputs and state"]
  node_dataset["Versioned dataset<br/>[dataset.py]"]
  node_export["Exports<br/>[export_service.py]"]
  node_models["Shared result models<br/>[analysis.py]"]
end

node_user(("Data analyst"))

node_user -->|"uses"| node_ui
node_ui -.->|"offers pages"| node_upload_ui
node_upload_ui -.->|"submits dataset"| node_dataset_service
node_dataset_service -.->|"loads data"| node_ingestion
node_ingestion -->|"normalizes into"| node_dataset
node_ui -->|"starts workflows"| node_workflow_service
node_workflow_service -->|"evaluates quality"| node_cleaning_service
node_cleaning_service -->|"profiles dataset"| node_profiling
node_cleaning_service -->|"evaluates issues"| node_quality
node_cleaning_ui -->|"runs pipeline"| node_workflow_service
node_cleaning_ui -->|"requests plan"| node_cleaning_service
node_cleaning_service -->|"builds plan"| node_cleaning_plan
node_cleaning_ui -->|"previews changes"| node_cleaning_service
node_cleaning_service -->|"generates preview"| node_cleaning_preview
node_cleaning_ui -->|"applies approvals"| node_cleaning_service
node_cleaning_service -->|"executes changes"| node_cleaning_executor
node_cleaning_service -->|"validates cleaning"| node_validation
node_cleaning_service -->|"derives version"| node_dataset
node_workflow_service -->|"detects domain and KPIs"| node_analytics
node_workflow_service -->|"builds dashboard plan"| node_dashboard
node_analytics -.->|"recommends and calculates"| node_kpi
node_user -->|"asks SQL questions"| node_sql
node_sql -->|"executes queries"| node_duckdb
node_user -->|"asks analyst"| node_ai
node_analytics -.->|"generates explanations"| node_ai
node_workflow_service -->|"tracks versions"| node_dataset
node_user -->|"requests exports"| node_export
node_export -.->|"exports dataset"| node_dataset
node_analytics -.->|"produces analysis results"| node_models

click node_ui "https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/app.py"
click node_upload_ui "https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/pages/02_Upload_Data.py"
click node_workflow_service "https://github.com/ahmednasser5/datapilot-ai/blob/main/services/workflow_service.py"
click node_dataset_service "https://github.com/ahmednasser5/datapilot-ai/blob/main/services/dataset_service.py"
click node_dataset "https://github.com/ahmednasser5/datapilot-ai/blob/main/models/dataset.py"
click node_ingestion "https://github.com/ahmednasser5/datapilot-ai/blob/main/ingestion/loader_registry.py"
click node_profiling "https://github.com/ahmednasser5/datapilot-ai/blob/main/profiling/profiler.py"
click node_quality "https://github.com/ahmednasser5/datapilot-ai/blob/main/quality/quality_engine.py"
click node_cleaning_ui "https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/pages/05_Cleaning_Studio.py"
click node_cleaning_service "https://github.com/ahmednasser5/datapilot-ai/blob/main/services/cleaning_service.py"
click node_cleaning_plan "https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/cleaning_planner.py"
click node_cleaning_preview "https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/preview.py"
click node_cleaning_executor "https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/cleaning_executor.py"
click node_validation "https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/cleaning_validator.py"
click node_sql "https://github.com/ahmednasser5/datapilot-ai/blob/main/sql/query_executor.py"
click node_duckdb "https://github.com/ahmednasser5/datapilot-ai/blob/main/sql/duckdb_engine.py"
click node_analytics "https://github.com/ahmednasser5/datapilot-ai/blob/main/services/analytics_service.py"
click node_kpi "https://github.com/ahmednasser5/datapilot-ai/blob/main/kpi/kpi_calculator.py"
click node_dashboard "https://github.com/ahmednasser5/datapilot-ai/blob/main/services/dashboard_service.py"
click node_ai "https://github.com/ahmednasser5/datapilot-ai/blob/main/ai/analyst_agent.py"
click node_export "https://github.com/ahmednasser5/datapilot-ai/blob/main/services/export_service.py"
click node_models "https://github.com/ahmednasser5/datapilot-ai/blob/main/models/analysis.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_upload_ui,node_cleaning_ui toneBlue
class node_workflow_service,node_dataset_service,node_cleaning_service toneAmber
class node_ingestion,node_profiling,node_quality,node_cleaning_plan,node_cleaning_preview,node_cleaning_executor,node_validation,node_sql,node_duckdb toneMint
class node_analytics,node_kpi,node_dashboard,node_ai toneRose
class node_dataset,node_export,node_models,node_user toneIndigo
```

## Components

### User experience

- [Streamlit interface](https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/app.py): `streamlit_app/app.py`
- [Upload and pages](https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/pages/02_Upload_Data.py): `streamlit_app/pages/02_Upload_Data.py`
- [Cleaning studio](https://github.com/ahmednasser5/datapilot-ai/blob/main/streamlit_app/pages/05_Cleaning_Studio.py): `streamlit_app/pages/05_Cleaning_Studio.py`

### Application workflow

- [Workflow service](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/workflow_service.py): `services/workflow_service.py`
- [Dataset service](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/dataset_service.py): `services/dataset_service.py`
- [Cleaning service](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/cleaning_service.py): `services/cleaning_service.py`

### Data engines

- [Format loaders](https://github.com/ahmednasser5/datapilot-ai/blob/main/ingestion/loader_registry.py): `ingestion/loader_registry.py`
- [Profiling](https://github.com/ahmednasser5/datapilot-ai/blob/main/profiling/profiler.py): `profiling/profiler.py`
- [Quality assessment](https://github.com/ahmednasser5/datapilot-ai/blob/main/quality/quality_engine.py): `quality/quality_engine.py`
- [Cleaning planner](https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/cleaning_planner.py): `cleaning/cleaning_planner.py`
- [Cleaning preview](https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/preview.py): `cleaning/preview.py`
- [Cleaning executor](https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/cleaning_executor.py): `cleaning/cleaning_executor.py`
- [Cleaning validation](https://github.com/ahmednasser5/datapilot-ai/blob/main/cleaning/cleaning_validator.py): `cleaning/cleaning_validator.py`
- [SQL analytics](https://github.com/ahmednasser5/datapilot-ai/blob/main/sql/query_executor.py): `sql/query_executor.py`
- [DuckDB engine](https://github.com/ahmednasser5/datapilot-ai/blob/main/sql/duckdb_engine.py): `sql/duckdb_engine.py`

### Analysis and AI

- [Analytics and AI](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/analytics_service.py): `services/analytics_service.py`
- [Domain KPIs](https://github.com/ahmednasser5/datapilot-ai/blob/main/kpi/kpi_calculator.py): `kpi/kpi_calculator.py`
- [Dashboard planning](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/dashboard_service.py): `services/dashboard_service.py`
- [AI reasoning](https://github.com/ahmednasser5/datapilot-ai/blob/main/ai/analyst_agent.py): `ai/analyst_agent.py`

### Outputs and state

- [Versioned dataset](https://github.com/ahmednasser5/datapilot-ai/blob/main/models/dataset.py): `models/dataset.py`
- [Exports](https://github.com/ahmednasser5/datapilot-ai/blob/main/services/export_service.py): `services/export_service.py`
- [Shared result models](https://github.com/ahmednasser5/datapilot-ai/blob/main/models/analysis.py): `models/analysis.py`

### Other components

- **Data analyst**

## Connections

- Data analyst → Streamlit interface: uses (evidence: `README.md`)
- Streamlit interface → Upload and pages: offers pages
- Upload and pages → Dataset service: submits dataset
- Dataset service → Format loaders: loads data
- Format loaders → Versioned dataset: normalizes into (evidence: `README.md`)
- Streamlit interface → Workflow service: starts workflows (evidence: `README.md`)
- Workflow service → Cleaning service: evaluates quality (evidence: `services/workflow_service.py`)
- Cleaning service → Profiling: profiles dataset (evidence: `services/cleaning_service.py`)
- Cleaning service → Quality assessment: evaluates issues (evidence: `services/cleaning_service.py`)
- Cleaning studio → Workflow service: runs pipeline (evidence: `streamlit_app/pages/05_Cleaning_Studio.py`)
- Cleaning studio → Cleaning service: requests plan (evidence: `streamlit_app/pages/05_Cleaning_Studio.py`)
- Cleaning service → Cleaning planner: builds plan (evidence: `services/cleaning_service.py`)
- Cleaning studio → Cleaning service: previews changes (evidence: `streamlit_app/pages/05_Cleaning_Studio.py`)
- Cleaning service → Cleaning preview: generates preview (evidence: `services/cleaning_service.py`)
- Cleaning studio → Cleaning service: applies approvals (evidence: `streamlit_app/pages/05_Cleaning_Studio.py`)
- Cleaning service → Cleaning executor: executes changes (evidence: `services/cleaning_service.py`)
- Cleaning service → Cleaning validation: validates cleaning (evidence: `services/cleaning_service.py`)
- Cleaning service → Versioned dataset: derives version (evidence: `services/cleaning_service.py`)
- Workflow service → Analytics and AI: detects domain and KPIs (evidence: `services/workflow_service.py`)
- Workflow service → Dashboard planning: builds dashboard plan (evidence: `services/workflow_service.py`)
- Analytics and AI → Domain KPIs: recommends and calculates
- Data analyst → SQL analytics: asks SQL questions (evidence: `README.md`)
- SQL analytics → DuckDB engine: executes queries (evidence: `README.md`)
- Data analyst → AI reasoning: asks analyst (evidence: `README.md`)
- Analytics and AI → AI reasoning: generates explanations (evidence: `services/analytics_service.py`)
- Workflow service → Versioned dataset: tracks versions (evidence: `services/workflow_service.py`)
- Data analyst → Exports: requests exports (evidence: `README.md`)
- Exports → Versioned dataset: exports dataset
- Analytics and AI → Shared result models: produces analysis results (evidence: `services/analytics_service.py`)
