How pvsatvika/data_forge works
Data Forge is a web platform for profiling uploaded datasets and producing reversible, validated cleaning results. The principal workflow runs from upload and profiling through AI-assisted recommendations, plan simulation and approval, execution, validation, and versioning; users can inspect history, download versions, and roll back. The frontend calls the FastAPI API, whose dataset route coordinates most backend stages. The graph combines README-documented capabilities with visible source and SOURCE INDEX references; unsampled implementations and some endpoint-level details are represented only where documented.
Main entry points: dataset.py Dataset API · service.py Semantic analysis · simulator.py Dry-run simulator · manager.py Version manager
Components 22
Web workflow
- Web app
frontend/src/App.tsx - Workflow pages
frontend/src/pages - API client
frontend/src/services/api.ts
API orchestration
- Dataset API
backend/app/api/routes/dataset.py
Analysis and planning
- Semantic analysis
backend/app/ai/service.py - Prompt builder
backend/app/ai/prompts.py - Groq client
backend/app/ai/client.py - AI response validation
backend/app/ai/validation.py - Cleaning plan builder
backend/app/planning/builder.py
Dataset processing
- Dataset profiler
backend/app/profiling/profiler.py - File validation
backend/app/security/file_validation.py - Dry-run simulator
backend/app/loss_estimation/simulator.py - Loss calculator
backend/app/loss_estimation/calculator.py - Allow-listed transforms
backend/app/transformations/operations.py
Validation and history
- Validation engine
backend/app/validation/engine.py - Version manager
backend/app/versioning/manager.py - Rollback engine
backend/app/rollback - SQLite metadata
backend/app/db/models.py - Versioned Parquet
backend/app/versioning/manager.py - Version download
backend/app/api/routes/dataset.py
Other components
- Data engineer
- Groq API
Connections 24
- Data engineer → Web app: uses
frontend/src/App.tsx - Web app → Workflow pages: routes to
frontend/src/App.tsx - Workflow pages → API client: requests
frontend/src/pages/AIAnalysis.tsx - API client → Dataset API: HTTP requests
frontend/src/services/api.ts - Dataset API → File validation: validates uploads
backend/app/api/routes/dataset.py - Dataset API → Dataset profiler: profiles dataset
backend/app/api/routes/dataset.py - Dataset API → SQLite metadata: stores metadata
backend/app/api/routes/dataset.py - Dataset API → Semantic analysis: requests analysis
backend/app/api/routes/dataset.py - Semantic analysis → Prompt builder: builds prompt
backend/app/ai/service.py - Semantic analysis → Groq client: submits prompt
backend/app/ai/service.py - Groq client → Groq API: sends request
backend/app/ai/client.py - Semantic analysis → AI response validation: validates response
backend/app/ai/service.py - Dataset API → Cleaning plan builder: builds plan
backend/app/api/routes/dataset.py - Dataset API → Dry-run simulator: simulates plan
backend/app/api/routes/dataset.py - Dry-run simulator → Allow-listed transforms: applies operations
backend/app/loss_estimation/simulator.py - Dry-run simulator → Loss calculator: calculates loss
backend/app/loss_estimation/simulator.py - Dataset API → Validation engine: validates execution
backend/app/api/routes/dataset.py - Dataset API → Allow-listed transforms: executes approved plan
backend/app/api/routes/dataset.py - Dataset API → Version manager: commits version
backend/app/api/routes/dataset.py - Version manager → SQLite metadata: records version
backend/app/versioning/manager.py - Version manager → Versioned Parquet: stores dataset state
backend/app/versioning/manager.py - Dataset API → Rollback engine: requests rollback
- Dataset API → Version download: serves downloads
backend/app/api/routes/dataset.py - Version download → Versioned Parquet: streams version