How murtaza-nasir/maestro works
Project Type and Purpose
• MAESTRO is a full-stack, self-hosted AI-powered research application.
• It provides both a Streamlit web UI and a CLI front-end to orchestrate a multi-agent research workflow, backed by a Retrieval-Augmented Generation (RAG) pipeline and a local vector store.
File Structure
• Top-level “ai_researcher” directory contains three main sub-packages:
• agentic_layer – orchestrates the multi-agent collaboration (Planning, Research, Reflection, Writing)
• core_rag – implements document ingestion, chunking, embedding, vector search, reranking, metadata extraction
• ui – hosts Streamlit/Gradio front-ends and file conversion utilities
• main_cli.py – entry point for non-interactive CLI tasks (ingest, query, run-research)
• Configuration files: .env.example, config.py, .streamlit/config.toml, Dockerfiles, docker-compose.yml
• Data stores under ai_researcher/data: raw PDFs, processed metadata (SQLite), vector embeddings (ChromaDB)
README for Additional Insights
• Multi-agent workflow: Planning → Research (local + web search) → Reflection → Writing
• RAG pipeline stages: PDF→Markdown conversion, metadata extraction, intelligent chunking, dense+sparse embeddings, vector store, hybrid retrieval, optional reranking
• Cost & usage tracking for LLMs and web search integrated into reports
• Two interaction modes: Streamlit UI (chat + mission control) and CLI (batch, scripted)
Main Components
a. Front-End Clients
• Web Browser ↔ Streamlit App (ai_researcher/ui/app.py)
• CLI Process (python -m ai_researcher.main_cli)
b. Application Layer (Back-End)
• UI Server (Streamlit/Gradio)
• CLI Controller
• Agentic Layer:
– Agent Controller & Context Manager
– Agents: PlanningAgent, ResearchAgent, ReflectionAgent, WritingAgent, plus utility/messenger agents
– Tool Registry & Tools (DocumentSearch, WebSearch, Calculator, PythonTool, FileReader, WebPageFetcher)
• RAG Layer:
– Document Ingestion: Chunker, MetadataExtractor, Embedder, Processor
– Vector Store (ChromaDB) & Processed-file tracker (SQLite)
– Retriever, QueryPreparer, QueryStrategist, Re-ranker
c. Data Stores & Persistence
• File System: Raw PDFs, converted Markdown, intermediate artifacts
• SQLite DB: ingest metadata, mission state, cost logs
• ChromaDB: dense + sparse embeddings, document metadata
d. External Services
• LLM Providers (OpenRouter API, local LLM endpoint)
• Web Search Providers (Tavily, LinkUp)
e. Output Artifacts
• Markdown research reports
• Cost & usage logs embedded in reports
Relationships and Data Flow
• UI/CLI → Agentic Layer (mission control calls)
• Agentic Layer ↔ Tools (for search, calculation, file read)
• ResearchAgent uses Core RAG to query Vector Store & Web Search → returns snippets
• Agents invoke LLM calls via Model Dispatcher → remote or local LLM endpoints
• Results and state updates flow back up to Controller → persisted in DB and returned to UI/CLI
• Document ingestion path: CLI “ingest” command → core_rag pipeline → SQLite + ChromaDB
Architectural Patterns & Design Principles
• Layered architecture: Presentation (UI/CLI), Orchestration (Agentic), Data (RAG), Persistence
• Modular plugin pattern: Tool Registry dynamically injects tools into agents
• Hybrid retrieval strategy: combines dense (embedding) + sparse (keyword) search with configurable weights
• Iterative, feedback-driven workflow: ReflectionAgent loops back to ResearchAgent for refinement
Technologies, Frameworks, and Libraries
• Python 3.x, Streamlit, Gradio
• marker (PDF→Markdown), pymupdf (metadata), torch (embeddings), ChromaDB
• dotenv for config, SQLite for metadata, docker/docker-compose for containerization
• LLM providers via OpenRouter SDK or local HTTP API
Diagramming Recommendations
• Blue rectangles for services/components (UI Server, CLI Service, Agentic Engine, RAG Engine)
• Red clouds for external services (LLM API, Web Search API)
• Interactions with protocol/technology notes (e.g., “HTTP REST to OpenRouter”, “Python gRPC to local LLM”).
Essential Details to Capture
• Both interactive (UI) and non-interactive (CLI) entry points feeding into the same orchestration layer
• Multi-agent roles and their collaboration via the Agent Controller
• Hybrid retrieval architecture and optional reranking stage
• Data persistence layers and cost-tracking integration
• External dependencies (LLM, web search) and fallback configuration