How bhagyesh-074/ml_pipeline_agent works
Autonomous ML Pipeline Agent accepts a tabular dataset and target, then validates it and runs EDA, feature engineering, model selection/tuning, and reporting. The primary interface is the Next.js web app, backed by FastAPI’s asynchronous run handling; a Streamlit interface is also present. Feature engineering and model selection use retrieved knowledge-base context, while reporting produces charts, Markdown/PDF, and a model artifact. The graph follows documented workflow and sampled calls; links for unsampled frontend/backend interactions rely on the README, and unshown implementation details are omitted.
Main entry points: orchestrator.py Pipeline Orchestrator · main.py FastAPI Service · feature_engineering_agent.py Feature Engineering · model_agent.py Model Selection
Components 29
User Interfaces
- Web App
frontend/app/page.tsx - Run Results Page
frontend/app/runs/[run_id]/page.tsx - Report View
frontend/components/ReportView.tsx - Stage Progress
frontend/components/StageProgress.tsx - Downloads
frontend/components/DownloadButtons.tsx - Streamlit App
app/streamlit_app.py - Streamlit Helpers
app/ui_helpers.py
Run Control
- FastAPI Service
api/main.py - Job Store
api/job_store.py - Pipeline Orchestrator
agents/orchestrator.py
ML Analysis
- Dataset Validation
tools/data_utils.py - Exploratory Analysis
agents/eda_agent.py - Dataset Profiling
tools/profiling.py - Feature Engineering
agents/feature_engineering_agent.py - Feature Code Generation
tools/feature_codegen.py - Feature Sandbox
tools/sandbox_executor.py - Apply Features
tools/feature_application.py - Model Selection
agents/model_agent.py - Training and Tuning
tools/model_training.py - Data Characteristics
tools/data_characteristics.py
Knowledge and LLM
- Knowledge Retrieval
rag/retriever.py - Knowledge Indexer
rag/embed_and_index.py - Chroma Collection
- Groq LLM Client
llm_client.py
Reporting and Artifacts
- Reporting Agent
agents/reporting_agent.py - Report Charts
tools/report_charts.py - Report Writer
tools/report_writer.py
Other components
- Dataset User
- Groq API
Connections 33
- Dataset User → Web App: uploads dataset
README.md - Web App → FastAPI Service: submits run
README.md - FastAPI Service → Pipeline Orchestrator: starts pipeline
api/main.py - FastAPI Service → Job Store: records and reads jobs
api/main.py - FastAPI Service → Dataset Validation: loads dataset
api/main.py - FastAPI Service → Run Results Page: serves run results
README.md - Run Results Page → Report View: renders report
README.md - Run Results Page → Stage Progress: shows progress
README.md - Run Results Page → Downloads: offers artifacts
README.md - Dataset User → Streamlit App: uploads dataset
app/streamlit_app.py - Streamlit App → Streamlit Helpers: uses helpers
app/streamlit_app.py - Streamlit App → Pipeline Orchestrator: runs pipeline
app/streamlit_app.py - Pipeline Orchestrator → Dataset Validation: validates and infers
agents/orchestrator.py - Pipeline Orchestrator → Exploratory Analysis: runs EDA
agents/orchestrator.py - Exploratory Analysis → Dataset Profiling: profiles data
agents/eda_agent.py - Pipeline Orchestrator → Feature Engineering: runs feature engineering
agents/orchestrator.py - Feature Engineering → Knowledge Retrieval: retrieves context
agents/feature_engineering_agent.py - Feature Engineering → Feature Code Generation: generates feature code
agents/feature_engineering_agent.py - Feature Engineering → Feature Sandbox: executes feature code
agents/feature_engineering_agent.py - Pipeline Orchestrator → Apply Features: applies kept features
agents/orchestrator.py - Pipeline Orchestrator → Model Selection: runs model selection
agents/orchestrator.py - Model Selection → Knowledge Retrieval: retrieves context
agents/model_agent.py - Model Selection → Data Characteristics: analyzes dataset
agents/model_agent.py - Model Selection → Training and Tuning: trains and tunes
agents/model_agent.py - Pipeline Orchestrator → Reporting Agent: runs reporting
agents/orchestrator.py - Reporting Agent → Report Charts: creates charts
agents/reporting_agent.py - Reporting Agent → Report Writer: writes report
agents/reporting_agent.py - Knowledge Retrieval → Knowledge Indexer: gets collection
rag/retriever.py - Knowledge Indexer → Chroma Collection: provides collection
- Feature Engineering → Groq LLM Client: requests suggestions
agents/feature_engineering_agent.py - Model Selection → Groq LLM Client: requests rationale
agents/model_agent.py - Exploratory Analysis → Groq LLM Client: requests analysis
agents/eda_agent.py - Groq LLM Client → Groq API: calls completions
llm_client.py