# pvsatvika/data_forge architecture

> Architecture diagram of the GitHub repository pvsatvika/data_forge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pvsatvika/data_forge
- Repository: https://github.com/pvsatvika/data_forge

## Overview

Data Forge is a web platform for profiling uploaded datasets and producing reversible, validated cleaning results. The principal workflow runs from upload and profiling through AI-assisted recommendations, plan simulation and approval, execution, validation, and versioning; users can inspect history, download versions, and roll back. The frontend calls the FastAPI API, whose dataset route coordinates most backend stages. The graph combines README-documented capabilities with visible source and SOURCE INDEX references; unsampled implementations and some endpoint-level details are represented only where documented.

Main entry points: [`backend/app/api/routes/dataset.py`](https://github.com/pvsatvika/data_forge/blob/main/backend/app/api/routes/dataset.py) (Dataset API), [`backend/app/ai/service.py`](https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/service.py) (Semantic analysis), [`backend/app/loss_estimation/simulator.py`](https://github.com/pvsatvika/data_forge/blob/main/backend/app/loss_estimation/simulator.py) (Dry-run simulator), [`backend/app/versioning/manager.py`](https://github.com/pvsatvika/data_forge/blob/main/backend/app/versioning/manager.py) (Version manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web workflow"]
  node_app["Web app<br/>[App.tsx]"]
  node_workflow_ui["Workflow pages"]
  node_api_client["API client<br/>[api.ts]"]
end

subgraph group_api["API orchestration"]
  node_routes["Dataset API<br/>[dataset.py]"]
end

subgraph group_intelligence["Analysis and planning"]
  node_semantic["Semantic analysis<br/>[service.py]"]
  node_prompt["Prompt builder<br/>[prompts.py]"]
  node_groq_client["Groq client<br/>[client.py]"]
  node_response_validation["AI response validation<br/>[validation.py]"]
  node_plan_builder["Cleaning plan builder<br/>[builder.py]"]
end

subgraph group_data["Dataset processing"]
  node_profiler["Dataset profiler<br/>[profiler.py]"]
  node_file_security["File validation<br/>[file_validation.py]"]
  node_simulator["Dry-run simulator<br/>[simulator.py]"]
  node_loss_calculator["Loss calculator<br/>[calculator.py]"]
  node_transformations["Allow-listed transforms<br/>[operations.py]"]
end

subgraph group_history["Validation and history"]
  node_validation_engine["Validation engine<br/>[engine.py]"]
  node_version_manager["Version manager<br/>[manager.py]"]
  node_rollback["Rollback engine"]
  node_metadata_store[("SQLite metadata<br/>[models.py]")]
  node_parquet_store[("Versioned Parquet<br/>[manager.py]")]
  node_download["Version download<br/>[dataset.py]"]
end

node_user(("Data engineer"))
node_groq{{"Groq API"}}

node_user -->|"uses"| node_app
node_app -->|"routes to"| node_workflow_ui
node_workflow_ui -->|"requests"| node_api_client
node_api_client -->|"HTTP requests"| node_routes
node_routes -->|"validates uploads"| node_file_security
node_routes -->|"profiles dataset"| node_profiler
node_routes -->|"stores metadata"| node_metadata_store
node_routes -->|"requests analysis"| node_semantic
node_semantic -->|"builds prompt"| node_prompt
node_semantic -->|"submits prompt"| node_groq_client
node_groq_client -->|"sends request"| node_groq
node_semantic -->|"validates response"| node_response_validation
node_routes -->|"builds plan"| node_plan_builder
node_routes -->|"simulates plan"| node_simulator
node_simulator -->|"applies operations"| node_transformations
node_simulator -->|"calculates loss"| node_loss_calculator
node_routes -->|"validates execution"| node_validation_engine
node_routes -->|"executes approved plan"| node_transformations
node_routes -->|"commits version"| node_version_manager
node_version_manager -->|"records version"| node_metadata_store
node_version_manager -->|"stores dataset state"| node_parquet_store
node_routes -->|"requests rollback"| node_rollback
node_routes -->|"serves downloads"| node_download
node_download -->|"streams version"| node_parquet_store

click node_app "https://github.com/pvsatvika/data_forge/blob/main/frontend/src/App.tsx"
click node_workflow_ui "https://github.com/pvsatvika/data_forge/tree/main/frontend/src/pages"
click node_api_client "https://github.com/pvsatvika/data_forge/blob/main/frontend/src/services/api.ts"
click node_routes "https://github.com/pvsatvika/data_forge/blob/main/backend/app/api/routes/dataset.py"
click node_profiler "https://github.com/pvsatvika/data_forge/blob/main/backend/app/profiling/profiler.py"
click node_file_security "https://github.com/pvsatvika/data_forge/blob/main/backend/app/security/file_validation.py"
click node_semantic "https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/service.py"
click node_prompt "https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/prompts.py"
click node_groq_client "https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/client.py"
click node_response_validation "https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/validation.py"
click node_plan_builder "https://github.com/pvsatvika/data_forge/blob/main/backend/app/planning/builder.py"
click node_simulator "https://github.com/pvsatvika/data_forge/blob/main/backend/app/loss_estimation/simulator.py"
click node_loss_calculator "https://github.com/pvsatvika/data_forge/blob/main/backend/app/loss_estimation/calculator.py"
click node_transformations "https://github.com/pvsatvika/data_forge/blob/main/backend/app/transformations/operations.py"
click node_validation_engine "https://github.com/pvsatvika/data_forge/blob/main/backend/app/validation/engine.py"
click node_version_manager "https://github.com/pvsatvika/data_forge/blob/main/backend/app/versioning/manager.py"
click node_rollback "https://github.com/pvsatvika/data_forge/tree/main/backend/app/rollback"
click node_metadata_store "https://github.com/pvsatvika/data_forge/blob/main/backend/app/db/models.py"
click node_parquet_store "https://github.com/pvsatvika/data_forge/blob/main/backend/app/versioning/manager.py"
click node_download "https://github.com/pvsatvika/data_forge/blob/main/backend/app/api/routes/dataset.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_workflow_ui,node_api_client toneBlue
class node_routes toneAmber
class node_semantic,node_prompt,node_groq_client,node_response_validation,node_plan_builder,node_groq toneMint
class node_profiler,node_file_security,node_simulator,node_loss_calculator,node_transformations toneRose
class node_validation_engine,node_version_manager,node_rollback,node_metadata_store,node_parquet_store,node_download,node_user toneIndigo
```

## Components

### Web workflow

- [Web app](https://github.com/pvsatvika/data_forge/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Workflow pages](https://github.com/pvsatvika/data_forge/tree/main/frontend/src/pages): `frontend/src/pages`
- [API client](https://github.com/pvsatvika/data_forge/blob/main/frontend/src/services/api.ts): `frontend/src/services/api.ts`

### API orchestration

- [Dataset API](https://github.com/pvsatvika/data_forge/blob/main/backend/app/api/routes/dataset.py): `backend/app/api/routes/dataset.py`

### Analysis and planning

- [Semantic analysis](https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/service.py): `backend/app/ai/service.py`
- [Prompt builder](https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/prompts.py): `backend/app/ai/prompts.py`
- [Groq client](https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/client.py): `backend/app/ai/client.py`
- [AI response validation](https://github.com/pvsatvika/data_forge/blob/main/backend/app/ai/validation.py): `backend/app/ai/validation.py`
- [Cleaning plan builder](https://github.com/pvsatvika/data_forge/blob/main/backend/app/planning/builder.py): `backend/app/planning/builder.py`

### Dataset processing

- [Dataset profiler](https://github.com/pvsatvika/data_forge/blob/main/backend/app/profiling/profiler.py): `backend/app/profiling/profiler.py`
- [File validation](https://github.com/pvsatvika/data_forge/blob/main/backend/app/security/file_validation.py): `backend/app/security/file_validation.py`
- [Dry-run simulator](https://github.com/pvsatvika/data_forge/blob/main/backend/app/loss_estimation/simulator.py): `backend/app/loss_estimation/simulator.py`
- [Loss calculator](https://github.com/pvsatvika/data_forge/blob/main/backend/app/loss_estimation/calculator.py): `backend/app/loss_estimation/calculator.py`
- [Allow-listed transforms](https://github.com/pvsatvika/data_forge/blob/main/backend/app/transformations/operations.py): `backend/app/transformations/operations.py`

### Validation and history

- [Validation engine](https://github.com/pvsatvika/data_forge/blob/main/backend/app/validation/engine.py): `backend/app/validation/engine.py`
- [Version manager](https://github.com/pvsatvika/data_forge/blob/main/backend/app/versioning/manager.py): `backend/app/versioning/manager.py`
- [Rollback engine](https://github.com/pvsatvika/data_forge/tree/main/backend/app/rollback): `backend/app/rollback`
- [SQLite metadata](https://github.com/pvsatvika/data_forge/blob/main/backend/app/db/models.py): `backend/app/db/models.py`
- [Versioned Parquet](https://github.com/pvsatvika/data_forge/blob/main/backend/app/versioning/manager.py): `backend/app/versioning/manager.py`
- [Version download](https://github.com/pvsatvika/data_forge/blob/main/backend/app/api/routes/dataset.py): `backend/app/api/routes/dataset.py`

### Other components

- **Data engineer**
- **Groq API**

## Connections

- Data engineer → Web app: uses (evidence: `frontend/src/App.tsx`)
- Web app → Workflow pages: routes to (evidence: `frontend/src/App.tsx`)
- Workflow pages → API client: requests (evidence: `frontend/src/pages/AIAnalysis.tsx`)
- API client → Dataset API: HTTP requests (evidence: `frontend/src/services/api.ts`)
- Dataset API → File validation: validates uploads (evidence: `backend/app/api/routes/dataset.py`)
- Dataset API → Dataset profiler: profiles dataset (evidence: `backend/app/api/routes/dataset.py`)
- Dataset API → SQLite metadata: stores metadata (evidence: `backend/app/api/routes/dataset.py`)
- Dataset API → Semantic analysis: requests analysis (evidence: `backend/app/api/routes/dataset.py`)
- Semantic analysis → Prompt builder: builds prompt (evidence: `backend/app/ai/service.py`)
- Semantic analysis → Groq client: submits prompt (evidence: `backend/app/ai/service.py`)
- Groq client → Groq API: sends request (evidence: `backend/app/ai/client.py`)
- Semantic analysis → AI response validation: validates response (evidence: `backend/app/ai/service.py`)
- Dataset API → Cleaning plan builder: builds plan (evidence: `backend/app/api/routes/dataset.py`)
- Dataset API → Dry-run simulator: simulates plan (evidence: `backend/app/api/routes/dataset.py`)
- Dry-run simulator → Allow-listed transforms: applies operations (evidence: `backend/app/loss_estimation/simulator.py`)
- Dry-run simulator → Loss calculator: calculates loss (evidence: `backend/app/loss_estimation/simulator.py`)
- Dataset API → Validation engine: validates execution (evidence: `backend/app/api/routes/dataset.py`)
- Dataset API → Allow-listed transforms: executes approved plan (evidence: `backend/app/api/routes/dataset.py`)
- Dataset API → Version manager: commits version (evidence: `backend/app/api/routes/dataset.py`)
- Version manager → SQLite metadata: records version (evidence: `backend/app/versioning/manager.py`)
- Version manager → Versioned Parquet: stores dataset state (evidence: `backend/app/versioning/manager.py`)
- Dataset API → Rollback engine: requests rollback
- Dataset API → Version download: serves downloads (evidence: `backend/app/api/routes/dataset.py`)
- Version download → Versioned Parquet: streams version