# bhagyesh-074/ml_pipeline_agent architecture

> Architecture diagram of the GitHub repository bhagyesh-074/ml_pipeline_agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bhagyesh-074/ml_pipeline_agent
- Repository: https://github.com/bhagyesh-074/ml_pipeline_agent

## Overview

Autonomous ML Pipeline Agent accepts a tabular dataset and target, then validates it and runs EDA, feature engineering, model selection/tuning, and reporting. The primary interface is the Next.js web app, backed by FastAPI’s asynchronous run handling; a Streamlit interface is also present. Feature engineering and model selection use retrieved knowledge-base context, while reporting produces charts, Markdown/PDF, and a model artifact. The graph follows documented workflow and sampled calls; links for unsampled frontend/backend interactions rely on the README, and unshown implementation details are omitted.

Main entry points: [`agents/orchestrator.py`](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/orchestrator.py) (Pipeline Orchestrator), [`api/main.py`](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/api/main.py) (FastAPI Service), [`agents/feature_engineering_agent.py`](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/feature_engineering_agent.py) (Feature Engineering), [`agents/model_agent.py`](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/model_agent.py) (Model Selection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_web_ui["Web App<br/>[page.tsx]"]
  node_run_page["Run Results Page<br/>[page.tsx]"]
  node_report_view["Report View<br/>[ReportView.tsx]"]
  node_progress_ui["Stage Progress<br/>[StageProgress.tsx]"]
  node_download_ui["Downloads"]
  node_streamlit_ui["Streamlit App<br/>[streamlit_app.py]"]
  node_streamlit_helpers["Streamlit Helpers<br/>[ui_helpers.py]"]
end

subgraph group_run_control["Run Control"]
  node_api["FastAPI Service<br/>[main.py]"]
  node_job_store[("Job Store<br/>[job_store.py]")]
  node_orchestrator["Pipeline Orchestrator<br/>[orchestrator.py]"]
end

subgraph group_analysis["ML Analysis"]
  node_dataset_utils["Dataset Validation<br/>[data_utils.py]"]
  node_eda["Exploratory Analysis<br/>[eda_agent.py]"]
  node_profiling["Dataset Profiling<br/>[profiling.py]"]
  node_feature_agent["Feature Engineering"]
  node_feature_codegen["Feature Code Generation<br/>[feature_codegen.py]"]
  node_sandbox["Feature Sandbox"]
  node_feature_apply["Apply Features"]
  node_model_agent["Model Selection<br/>[model_agent.py]"]
  node_model_training["Training and Tuning<br/>[model_training.py]"]
  node_characteristics["Data Characteristics"]
end

subgraph group_knowledge["Knowledge and LLM"]
  node_retriever["Knowledge Retrieval<br/>[retriever.py]"]
  node_indexer["Knowledge Indexer<br/>[embed_and_index.py]"]
  node_vector_store[("Chroma Collection")]
  node_llm_client["Groq LLM Client<br/>[llm_client.py]"]
end

subgraph group_outputs["Reporting and Artifacts"]
  node_report_agent["Reporting Agent<br/>[reporting_agent.py]"]
  node_charts["Report Charts<br/>[report_charts.py]"]
  node_writer["Report Writer<br/>[report_writer.py]"]
end

node_user(("Dataset User"))
node_groq(("Groq API"))

node_user -->|"uploads dataset"| node_web_ui
node_web_ui -->|"submits run"| node_api
node_api -->|"starts pipeline"| node_orchestrator
node_api -->|"records and reads jobs"| node_job_store
node_api -->|"loads dataset"| node_dataset_utils
node_api -.->|"serves run results"| node_run_page
node_run_page -.->|"renders report"| node_report_view
node_run_page -.->|"shows progress"| node_progress_ui
node_run_page -.->|"offers artifacts"| node_download_ui
node_user -->|"uploads dataset"| node_streamlit_ui
node_streamlit_ui -->|"uses helpers"| node_streamlit_helpers
node_streamlit_ui -->|"runs pipeline"| node_orchestrator
node_orchestrator -->|"validates and infers"| node_dataset_utils
node_orchestrator -->|"runs EDA"| node_eda
node_eda -->|"profiles data"| node_profiling
node_orchestrator -->|"runs feature engineering"| node_feature_agent
node_feature_agent -->|"retrieves context"| node_retriever
node_feature_agent -->|"generates feature code"| node_feature_codegen
node_feature_agent -->|"executes feature code"| node_sandbox
node_orchestrator -->|"applies kept features"| node_feature_apply
node_orchestrator -->|"runs model selection"| node_model_agent
node_model_agent -->|"retrieves context"| node_retriever
node_model_agent -->|"analyzes dataset"| node_characteristics
node_model_agent -->|"trains and tunes"| node_model_training
node_orchestrator -->|"runs reporting"| node_report_agent
node_report_agent -->|"creates charts"| node_charts
node_report_agent -->|"writes report"| node_writer
node_retriever -->|"gets collection"| node_indexer
node_indexer -.->|"provides collection"| node_vector_store
node_feature_agent -.->|"requests suggestions"| node_llm_client
node_model_agent -->|"requests rationale"| node_llm_client
node_eda -->|"requests analysis"| node_llm_client
node_llm_client -->|"calls completions"| node_groq

click node_web_ui "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/app/page.tsx"
click node_run_page "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/app/runs/%5Brun_id%5D/page.tsx"
click node_report_view "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/components/ReportView.tsx"
click node_progress_ui "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/components/StageProgress.tsx"
click node_download_ui "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/components/DownloadButtons.tsx"
click node_streamlit_ui "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/app/streamlit_app.py"
click node_streamlit_helpers "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/app/ui_helpers.py"
click node_api "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/api/main.py"
click node_job_store "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/api/job_store.py"
click node_orchestrator "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/orchestrator.py"
click node_dataset_utils "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/data_utils.py"
click node_eda "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/eda_agent.py"
click node_profiling "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/profiling.py"
click node_feature_agent "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/feature_engineering_agent.py"
click node_feature_codegen "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/feature_codegen.py"
click node_sandbox "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/sandbox_executor.py"
click node_feature_apply "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/feature_application.py"
click node_model_agent "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/model_agent.py"
click node_model_training "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/model_training.py"
click node_characteristics "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/data_characteristics.py"
click node_report_agent "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/reporting_agent.py"
click node_charts "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/report_charts.py"
click node_writer "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/report_writer.py"
click node_retriever "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/rag/retriever.py"
click node_indexer "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/rag/embed_and_index.py"
click node_llm_client "https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/llm_client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web_ui,node_run_page,node_report_view,node_progress_ui,node_download_ui,node_streamlit_ui,node_streamlit_helpers,node_user toneBlue
class node_api,node_job_store,node_orchestrator toneAmber
class node_dataset_utils,node_eda,node_profiling,node_feature_agent,node_feature_codegen,node_sandbox,node_feature_apply,node_model_agent,node_model_training,node_characteristics,node_groq toneMint
class node_retriever,node_indexer,node_vector_store,node_llm_client toneRose
class node_report_agent,node_charts,node_writer toneIndigo
```

## Components

### User Interfaces

- [Web App](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/app/page.tsx): `frontend/app/page.tsx`
- [Run Results Page](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/app/runs/%5Brun_id%5D/page.tsx): `frontend/app/runs/[run_id]/page.tsx`
- [Report View](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/components/ReportView.tsx): `frontend/components/ReportView.tsx`
- [Stage Progress](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/components/StageProgress.tsx): `frontend/components/StageProgress.tsx`
- [Downloads](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/frontend/components/DownloadButtons.tsx): `frontend/components/DownloadButtons.tsx`
- [Streamlit App](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/app/streamlit_app.py): `app/streamlit_app.py`
- [Streamlit Helpers](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/app/ui_helpers.py): `app/ui_helpers.py`

### Run Control

- [FastAPI Service](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/api/main.py): `api/main.py`
- [Job Store](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/api/job_store.py): `api/job_store.py`
- [Pipeline Orchestrator](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/orchestrator.py): `agents/orchestrator.py`

### ML Analysis

- [Dataset Validation](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/data_utils.py): `tools/data_utils.py`
- [Exploratory Analysis](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/eda_agent.py): `agents/eda_agent.py`
- [Dataset Profiling](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/profiling.py): `tools/profiling.py`
- [Feature Engineering](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/feature_engineering_agent.py): `agents/feature_engineering_agent.py`
- [Feature Code Generation](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/feature_codegen.py): `tools/feature_codegen.py`
- [Feature Sandbox](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/sandbox_executor.py): `tools/sandbox_executor.py`
- [Apply Features](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/feature_application.py): `tools/feature_application.py`
- [Model Selection](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/model_agent.py): `agents/model_agent.py`
- [Training and Tuning](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/model_training.py): `tools/model_training.py`
- [Data Characteristics](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/data_characteristics.py): `tools/data_characteristics.py`

### Knowledge and LLM

- [Knowledge Retrieval](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/rag/retriever.py): `rag/retriever.py`
- [Knowledge Indexer](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/rag/embed_and_index.py): `rag/embed_and_index.py`
- **Chroma Collection**
- [Groq LLM Client](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/llm_client.py): `llm_client.py`

### Reporting and Artifacts

- [Reporting Agent](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/agents/reporting_agent.py): `agents/reporting_agent.py`
- [Report Charts](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/report_charts.py): `tools/report_charts.py`
- [Report Writer](https://github.com/bhagyesh-074/ml_pipeline_agent/blob/main/tools/report_writer.py): `tools/report_writer.py`

### Other components

- **Dataset User**
- **Groq API**

## Connections

- Dataset User → Web App: uploads dataset (evidence: `README.md`)
- Web App → FastAPI Service: submits run (evidence: `README.md`)
- FastAPI Service → Pipeline Orchestrator: starts pipeline (evidence: `api/main.py`)
- FastAPI Service → Job Store: records and reads jobs (evidence: `api/main.py`)
- FastAPI Service → Dataset Validation: loads dataset (evidence: `api/main.py`)
- FastAPI Service → Run Results Page: serves run results (evidence: `README.md`)
- Run Results Page → Report View: renders report (evidence: `README.md`)
- Run Results Page → Stage Progress: shows progress (evidence: `README.md`)
- Run Results Page → Downloads: offers artifacts (evidence: `README.md`)
- Dataset User → Streamlit App: uploads dataset (evidence: `app/streamlit_app.py`)
- Streamlit App → Streamlit Helpers: uses helpers (evidence: `app/streamlit_app.py`)
- Streamlit App → Pipeline Orchestrator: runs pipeline (evidence: `app/streamlit_app.py`)
- Pipeline Orchestrator → Dataset Validation: validates and infers (evidence: `agents/orchestrator.py`)
- Pipeline Orchestrator → Exploratory Analysis: runs EDA (evidence: `agents/orchestrator.py`)
- Exploratory Analysis → Dataset Profiling: profiles data (evidence: `agents/eda_agent.py`)
- Pipeline Orchestrator → Feature Engineering: runs feature engineering (evidence: `agents/orchestrator.py`)
- Feature Engineering → Knowledge Retrieval: retrieves context (evidence: `agents/feature_engineering_agent.py`)
- Feature Engineering → Feature Code Generation: generates feature code (evidence: `agents/feature_engineering_agent.py`)
- Feature Engineering → Feature Sandbox: executes feature code (evidence: `agents/feature_engineering_agent.py`)
- Pipeline Orchestrator → Apply Features: applies kept features (evidence: `agents/orchestrator.py`)
- Pipeline Orchestrator → Model Selection: runs model selection (evidence: `agents/orchestrator.py`)
- Model Selection → Knowledge Retrieval: retrieves context (evidence: `agents/model_agent.py`)
- Model Selection → Data Characteristics: analyzes dataset (evidence: `agents/model_agent.py`)
- Model Selection → Training and Tuning: trains and tunes (evidence: `agents/model_agent.py`)
- Pipeline Orchestrator → Reporting Agent: runs reporting (evidence: `agents/orchestrator.py`)
- Reporting Agent → Report Charts: creates charts (evidence: `agents/reporting_agent.py`)
- Reporting Agent → Report Writer: writes report (evidence: `agents/reporting_agent.py`)
- Knowledge Retrieval → Knowledge Indexer: gets collection (evidence: `rag/retriever.py`)
- Knowledge Indexer → Chroma Collection: provides collection
- Feature Engineering → Groq LLM Client: requests suggestions (evidence: `agents/feature_engineering_agent.py`)
- Model Selection → Groq LLM Client: requests rationale (evidence: `agents/model_agent.py`)
- Exploratory Analysis → Groq LLM Client: requests analysis (evidence: `agents/eda_agent.py`)
- Groq LLM Client → Groq API: calls completions (evidence: `llm_client.py`)
