# mouleshgs/autonomous-data-engineering architecture

> Architecture diagram of the GitHub repository mouleshgs/autonomous-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mouleshgs/autonomous-data-engineering
- Repository: https://github.com/mouleshgs/autonomous-data-engineering

## Overview

Orbital is an autonomous data-engineering MVP: users upload datasets, inspect profiles and quality, run an adaptive cleaning pipeline, and review processed output, decisions, and analytics. The documented pipeline runs through a LangGraph workflow; its stages load and profile data, plan safe operations, clean and validate, then persist results and optionally benchmark a selected model. The graph and tool excerpts support the workflow and data operations; several dashboard/API connections and optional integrations are documented or only partly visible, so uncertain links are marked dashed or omitted.

Main entry points: [`backend/app/graph/agent_graph.py`](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py) (LangGraph Workflow), [`frontend/src/App.tsx`](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/frontend/src/App.tsx) (React Dashboard), [`backend/app/main.py`](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/main.py) (HTTP API), [`backend/app/core.py`](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/core.py) (Runtime Registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User Experience"]
  node_dashboard["React Dashboard<br/>[App.tsx]"]
end

subgraph group_runtime["API Runtime"]
  node_api["HTTP API<br/>[main.py]"]
  node_core[("Runtime Registry<br/>[core.py]")]
  node_state["Pipeline State<br/>[pipeline_state.py]"]
end

subgraph group_pipeline["Data Pipeline"]
  node_graph["LangGraph Workflow<br/>[agent_graph.py]"]
  node_source["Source Analysis<br/>[agent_graph.py]"]
  node_profiler["Data Profiling<br/>[agent_graph.py]"]
  node_planner["Adaptive Planning<br/>[agent_graph.py]"]
  node_cleaning["Cleaning and Validation<br/>[agent_graph.py]"]
  node_transform["Output Transformation<br/>[agent_graph.py]"]
  node_benchmark["Model Benchmark<br/>[model_benchmark.py]"]
end

subgraph group_operations["Data Operations"]
  node_tools["Deterministic Data Tools<br/>[data_tools.py]"]
  node_agentbase["Agent Logging<br/>[base.py]"]
  node_processed["Processed CSV<br/>[storage.py]"]
end

node_user(("Dataset User"))
node_ollama(("Ollama Model"))

node_user -->|"uploads and runs"| node_dashboard
node_dashboard -.->|"requests"| node_api
node_api -->|"uses runtime"| node_core
node_core -->|"runs pipeline"| node_graph
node_graph -->|"dispatches"| node_source
node_graph -->|"dispatches"| node_profiler
node_graph -->|"dispatches"| node_planner
node_graph -->|"dispatches"| node_cleaning
node_graph -->|"dispatches"| node_transform
node_graph -.->|"optionally benchmarks"| node_benchmark
node_graph -->|"calls data tools"| node_tools
node_graph -->|"updates state"| node_state
node_transform -->|"persists CSV"| node_processed
node_planner -.->|"optionally plans with"| node_ollama
node_source -.->|"records decisions"| node_agentbase

click node_dashboard "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/frontend/src/App.tsx"
click node_api "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/main.py"
click node_core "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/core.py"
click node_state "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/schemas/pipeline_state.py"
click node_graph "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py"
click node_source "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py"
click node_profiler "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py"
click node_planner "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py"
click node_cleaning "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py"
click node_transform "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py"
click node_benchmark "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/agents/model_benchmark.py"
click node_tools "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/tools/data_tools.py"
click node_agentbase "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/agents/base.py"
click node_processed "https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/agents/storage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_user toneBlue
class node_api,node_core,node_state toneAmber
class node_graph,node_source,node_profiler,node_planner,node_cleaning,node_transform,node_benchmark toneMint
class node_tools,node_agentbase,node_processed toneRose
class node_ollama toneIndigo
```

## Components

### User Experience

- [React Dashboard](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`

### API Runtime

- [HTTP API](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/main.py): `backend/app/main.py`
- [Runtime Registry](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/core.py): `backend/app/core.py`
- [Pipeline State](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/schemas/pipeline_state.py): `backend/app/schemas/pipeline_state.py`

### Data Pipeline

- [LangGraph Workflow](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py): `backend/app/graph/agent_graph.py`
- [Source Analysis](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py): `backend/app/graph/agent_graph.py`
- [Data Profiling](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py): `backend/app/graph/agent_graph.py`
- [Adaptive Planning](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py): `backend/app/graph/agent_graph.py`
- [Cleaning and Validation](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py): `backend/app/graph/agent_graph.py`
- [Output Transformation](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/graph/agent_graph.py): `backend/app/graph/agent_graph.py`
- [Model Benchmark](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/agents/model_benchmark.py): `backend/app/agents/model_benchmark.py`

### Data Operations

- [Deterministic Data Tools](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/tools/data_tools.py): `backend/app/tools/data_tools.py`
- [Agent Logging](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/agents/base.py): `backend/app/agents/base.py`
- [Processed CSV](https://github.com/mouleshgs/autonomous-data-engineering/blob/main/backend/app/agents/storage.py): `backend/app/agents/storage.py`

### Other components

- **Dataset User**
- **Ollama Model**

## Connections

- Dataset User → React Dashboard: uploads and runs (evidence: `README.md`)
- React Dashboard → HTTP API: requests
- HTTP API → Runtime Registry: uses runtime (evidence: `backend/app/main.py`)
- Runtime Registry → LangGraph Workflow: runs pipeline (evidence: `backend/app/core.py`)
- LangGraph Workflow → Source Analysis: dispatches (evidence: `backend/app/graph/agent_graph.py`)
- LangGraph Workflow → Data Profiling: dispatches (evidence: `README.md`)
- LangGraph Workflow → Adaptive Planning: dispatches (evidence: `README.md`)
- LangGraph Workflow → Cleaning and Validation: dispatches (evidence: `README.md`)
- LangGraph Workflow → Output Transformation: dispatches (evidence: `README.md`)
- LangGraph Workflow → Model Benchmark: optionally benchmarks
- LangGraph Workflow → Deterministic Data Tools: calls data tools (evidence: `backend/app/graph/agent_graph.py`)
- LangGraph Workflow → Pipeline State: updates state (evidence: `backend/app/graph/agent_graph.py`)
- Output Transformation → Processed CSV: persists CSV (evidence: `backend/app/graph/agent_graph.py`)
- Adaptive Planning → Ollama Model: optionally plans with (evidence: `README.md`)
- Source Analysis → Agent Logging: records decisions
