# murtaza-nasir/maestro architecture

> Architecture diagram of the GitHub repository murtaza-nasir/maestro, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/murtaza-nasir/maestro
- Repository: https://github.com/murtaza-nasir/maestro

## Overview

##### Project Type and Purpose
- MAESTRO is a full-stack, self-hosted AI-powered research application.
- It provides both a Streamlit web UI and a CLI front-end to orchestrate a multi-agent research workflow, backed by a Retrieval-Augmented Generation (RAG) pipeline and a local vector store.

##### File Structure
- Top-level “ai_researcher” directory contains three main sub-packages:
- agentic_layer – orchestrates the multi-agent collaboration (Planning, Research, Reflection, Writing)
- core_rag – implements document ingestion, chunking, embedding, vector search, reranking, metadata extraction
- ui – hosts Streamlit/Gradio front-ends and file conversion utilities
- main_cli.py – entry point for non-interactive CLI tasks (ingest, query, run-research)
- Configuration files: `.env.example`, `config.py`, `.streamlit/config.toml`, Dockerfiles, docker-compose.yml
- Data stores under ai_researcher/data: raw PDFs, processed metadata (SQLite), vector embeddings (ChromaDB)

##### README for Additional Insights
- Multi-agent workflow: Planning → Research (local + web search) → Reflection → Writing
- RAG pipeline stages: PDF→Markdown conversion, metadata extraction, intelligent chunking, dense+sparse embeddings, vector store, hybrid retrieval, optional reranking
- Cost & usage tracking for LLMs and web search integrated into reports
- Two interaction modes: Streamlit UI (chat + mission control) and CLI (batch, scripted)

##### Main Components
a. Front-End Clients
- Web Browser ↔ Streamlit App (`ai_researcher/ui/app.py`)
- CLI Process (`python -m ai_researcher.main_cli`)
b. Application Layer (Back-End)
- UI Server (Streamlit/Gradio)
- CLI Controller
- Agentic Layer:
– Agent Controller & Context Manager
– Agents: PlanningAgent, ResearchAgent, ReflectionAgent, WritingAgent, plus utility/messenger agents
– Tool Registry & Tools (DocumentSearch, WebSearch, Calculator, PythonTool, FileReader, WebPageFetcher)
- RAG Layer:
– Document Ingestion: Chunker, MetadataExtractor, Embedder, Processor
– Vector Store (ChromaDB) & Processed-file tracker (SQLite)
– Retriever, QueryPreparer, QueryStrategist, Re-ranker
c. Data Stores & Persistence
- File System: Raw PDFs, converted Markdown, intermediate artifacts
- SQLite DB: ingest metadata, mission state, cost logs
- ChromaDB: dense + sparse embeddings, document metadata
d. External Services
- LLM Providers (OpenRouter API, local LLM endpoint)
- Web Search Providers (Tavily, LinkUp)
e. Output Artifacts
- Markdown research reports
- Cost & usage logs embedded in reports

##### Relationships and Data Flow
- UI/CLI → Agentic Layer (mission control calls)
- Agentic Layer ↔ Tools (for search, calculation, file read)
- ResearchAgent uses Core RAG to query Vector Store & Web Search → returns snippets
- Agents invoke LLM calls via Model Dispatcher → remote or local LLM endpoints
- Results and state updates flow back up to Controller → persisted in DB and returned to UI/CLI
- Document ingestion path: CLI “ingest” command → core_rag pipeline → SQLite + ChromaDB

##### Architectural Patterns & Design Principles
- Layered architecture: Presentation (UI/CLI), Orchestration (Agentic), Data (RAG), Persistence
- Modular plugin pattern: Tool Registry dynamically injects tools into agents
- Hybrid retrieval strategy: combines dense (embedding) + sparse (keyword) search with configurable weights
- Iterative, feedback-driven workflow: ReflectionAgent loops back to ResearchAgent for refinement

##### Technologies, Frameworks, and Libraries
- Python 3.x, Streamlit, Gradio
- marker (PDF→Markdown), pymupdf (metadata), torch (embeddings), ChromaDB
- dotenv for config, SQLite for metadata, docker/docker-compose for containerization
- LLM providers via OpenRouter SDK or local HTTP API

##### Diagramming Recommendations
- Blue rectangles for services/components (UI Server, CLI Service, Agentic Engine, RAG Engine)
- Red clouds for external services (LLM API, Web Search API)
- Interactions with protocol/technology notes (e.g., “HTTP REST to OpenRouter”, “Python gRPC to local LLM”).

##### Essential Details to Capture
- Both interactive (UI) and non-interactive (CLI) entry points feeding into the same orchestration layer
- Multi-agent roles and their collaboration via the Agent Controller
- Hybrid retrieval architecture and optional reranking stage
- Data persistence layers and cost-tracking integration
- External dependencies (LLM, web search) and fallback configuration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        WebBrowser["Web Browser"]:::external
        StreamlitUI["Streamlit App"]:::service
        GradioUI["Gradio App"]:::service
        CLIService["CLI Service"]:::service
    end

    WebBrowser -->|"HTTP"| StreamlitUI
    WebBrowser -->|"HTTP"| GradioUI
    CLIService -->|"Python CLI"| AgentController

    click StreamlitUI "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/ui/app.py"
    click GradioUI "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/ui/gradio_app.py"
    click CLIService "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/main_cli.py"

    %% Agentic Layer
    subgraph "Agentic Layer"
        direction TB
        AgentController["Agent Controller"]:::service
        ContextManager["Context Manager"]:::service
        ModelDispatcher["Model Dispatcher"]:::service
        ToolRegistry["Tool Registry"]:::service

        subgraph "Agents"
            direction TB
            PlanningAgent["PlanningAgent"]:::service
            ResearchAgent["ResearchAgent"]:::service
            ReflectionAgent["ReflectionAgent"]:::service
            WritingAgent["WritingAgent"]:::service
            WritingReflectionAgent["Writing-ReflectionAgent"]:::service
            MessengerAgent["MessengerAgent"]:::service
            NoteAssignmentAgent["NoteAssignmentAgent"]:::service
        end

        subgraph "Controllers & Managers"
            direction TB
            CoreController["Core Controller"]:::service
            ResearchManager["Research Manager"]:::service
            ReflectionManager["Reflection Manager"]:::service
            WritingManager["Writing Manager"]:::service
            ReportGenerator["Report Generator"]:::service
            UserInteraction["User Interaction Handler"]:::service
        end

        subgraph "Tools"
            direction TB
            DocumentSearch["DocumentSearch Tool"]:::service
            WebSearch["WebSearch Tool"]:::service
            WebPageFetcher["WebPageFetcher Tool"]:::service
            FileReader["FileReader Tool"]:::service
            PythonTool["PythonTool"]:::service
            CalculatorTool["Calculator Tool"]:::service
        end
    end

    StreamlitUI -->|"Mission calls"| AgentController
    GradioUI -->|"Mission calls"| AgentController

    AgentController --> ContextManager
    AgentController --> ModelDispatcher
    AgentController --> ToolRegistry
    AgentController --> PlanningAgent
    AgentController --> ResearchAgent
    AgentController --> ReflectionAgent
    AgentController --> WritingAgent

    ResearchAgent --> DocumentSearch
    ResearchAgent --> WebSearch
    ResearchAgent --> WebPageFetcher
    WritingAgent --> FileReader
    WritingAgent --> PythonTool
    WritingAgent --> CalculatorTool

    click AgentController "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agent_controller.py"
    click ContextManager "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/context_manager.py"
    click ModelDispatcher "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/model_dispatcher.py"
    click ToolRegistry "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tool_registry.py"

    click PlanningAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/planning_agent.py"
    click ResearchAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/research_agent.py"
    click ReflectionAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/reflection_agent.py"
    click WritingAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/writing_agent.py"
    click WritingReflectionAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/writing_reflection_agent.py"
    click MessengerAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/messenger_agent.py"
    click NoteAssignmentAgent "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/agents/note_assignment_agent.py"

    click CoreController "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/controller/core_controller.py"
    click ResearchManager "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/controller/research_manager.py"
    click ReflectionManager "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/controller/reflection_manager.py"
    click WritingManager "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/controller/writing_manager.py"
    click ReportGenerator "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/controller/report_generator.py"
    click UserInteraction "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/controller/user_interaction.py"

    click DocumentSearch "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tools/document_search.py"
    click WebSearch "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tools/web_search_tool.py"
    click WebPageFetcher "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tools/web_page_fetcher_tool.py"
    click FileReader "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tools/file_reader_tool.py"
    click PythonTool "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tools/python_tool.py"
    click CalculatorTool "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/agentic_layer/tools/calculator_tool.py"

    %% RAG Layer
    subgraph "RAG Layer"
        direction TB
        subgraph "Ingestion & Embedding"
            direction TB
            Chunker["Chunker"]:::service
            MetadataExtractor["MetadataExtractor"]:::service
            Embedder["Embedder"]:::service
            Processor["Processor"]:::service
        end
        subgraph "Retrieval & Ranking"
            direction TB
            Retriever["Retriever"]:::service
            QueryPreparer["QueryPreparer"]:::service
            QueryStrategist["QueryStrategist"]:::service
            Reranker["Reranker"]:::service
        end
        VectorStore["Vector Store (ChromaDB)"]:::db
        MetadataDB["SQLite Metadata DB"]:::db
    end

    click Chunker "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/chunker.py"
    click MetadataExtractor "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/metadata_extractor.py"
    click Embedder "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/embedder.py"
    click Processor "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/processor.py"

    click VectorStore "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/vector_store.py"
    click MetadataDB "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/database.py"
    click Retriever "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/retriever.py"
    click QueryPreparer "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/query_preparer.py"
    click QueryStrategist "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/query_strategist.py"
    click Reranker "https://github.com/murtaza-nasir/maestro/blob/main/ai_researcher/core_rag/reranker.py"

    ResearchAgent --> Retriever
    Retriever --> VectorStore
    Retriever --> MetadataDB
    Retriever --> Reranker
    Retriever --> QueryPreparer
    QueryPreparer --> QueryStrategist
    QueryStrategist --> Reranker

    %% Persistence Layer
    subgraph "Persistence & Storage"
        direction TB
        FileSystem["File System"]:::db
        ChromaDB["ChromaDB Store"]:::db
        SQLiteDB["SQLite DB"]:::db
    end

    Chunker --> FileSystem
    MetadataExtractor --> SQLiteDB
    Embedder --> ChromaDB
    Processor --> SQLiteDB

    VectorStore --> ChromaDB
    MetadataDB --> SQLiteDB

    %% External Services
    subgraph "External Services"
        direction TB
        LLMProviders["LLM Providers"]:::external
        WebSearchProviders["Web Search Providers"]:::external
    end

    ModelDispatcher --> LLMProviders
    WebSearch --> WebSearchProviders

    %% Output Artifacts
    subgraph "Output Artifacts"
        direction TB
        Reports["Markdown Reports"]:::artifact
        Logs["Cost & Usage Logs"]:::artifact
    end

    WritingAgent --> Reports
    AgentController --> Logs

    classDef service fill:#ADD8E6,stroke:#000,color:#000
    classDef db fill:#FFFF99,stroke:#000,color:#000
    classDef external fill:#FFB6C1,stroke:#000,color:#000
    classDef artifact fill:#98FB98,stroke:#000,color:#000
```
