# microsoft/mmctagent architecture

> Architecture diagram of the GitHub repository microsoft/mmctagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/mmctagent
- Repository: https://github.com/microsoft/mmctagent

## Overview

##### Project Type and Purpose
- MMCTAgent is a modular, full-stack AI framework for multi-modal critical thinking and complex visual reasoning.
- It provides:
- ImageAgent (static image Q&A with planner/critic loop)
- VideoAgent (video ingestion, indexing, and Q&A pipelines)
- A FastAPI front-end (‘app’) exposing ingestion and query endpoints
- An optional MCP server microservice for orchestrating planner/critic loops
- It is research-oriented, vendor-agnostic (pluggable provider system), and deployable to Azure via Docker/ARM templates.

##### File-Structure Analysis
Top-level directories:
── infra/ (ARM templates + Bash scripts for Azure infra: Event Hub, AI Search, OpenAI, Blob Storage, App Services)
── app/ (FastAPI microservice: ingestion_consumer, main API, routers, schemas, services, utilities)
── mcp_server/ (standalone TCP/HTTP server + client for planner/critic orchestration)
── mmct/ (core Python library)
- image_pipeline/ (ImageAgent entrypoint + tools/models)
- video_pipeline/ (IngestionPipeline + VideoAgent + keyframes, chapter generation, transcription)
- providers/ (factory + base + azure/openai/custom adapters for LLM, search, storage, vision, speech)
- config/, security/, utils/, tests/
── examples/, README files, Dockerfiles, CI support files.

##### Key Insights from README
- Two specialized agents (ImageAgent & VideoAgent) implementing “planner → critic → refine” loops
- Multi-cloud provider abstraction (Azure + OpenAI + local FAISS)
- FASTAPI “app” runs on Docker containers, connects to Azure Event Hub for ingestion, Azure AI Search / FAISS for vector DB, OpenAI/Azure OpenAI for LLM, Blob Storage for assets.
- infra/ contains deployment scripts and ARM templates for all needed Azure services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        CLI["CLI / Jupyter / Examples"]:::external
        WebUI["Web UI / HTTP Client"]:::external
    end

    %% API Layer
    subgraph "API Layer (FastAPI app)"
        API["FastAPI App"]:::internal
        IngestRouter["/ingest Router"]:::internal
        QueryRouter["/query Router"]:::internal
    end

    %% Ingestion Consumer
    subgraph "Ingestion Consumer"
        IngestCons["IngestionConsumer Worker"]:::internal
    end

    %% Domain Layer
    subgraph "Domain Layer (mmct core)"
        subgraph "Agents & Pipelines"
            ImageAgent["ImageAgent"]:::core
            VideoAgent["VideoAgent"]:::core
            IngestPipeline["IngestionPipeline"]:::core
        end
        ProviderFactory["Provider Factory"]:::core
    end

    %% MCP Microservice
    subgraph "MCP Microservice"
        MCP["MCP Server"]:::internal
    end

    %% External Services
    subgraph "External Services"
        EH["Azure Event Hub"]:::external
        SearchDB["Azure AI Search / FAISS"]:::external
        LLM["Azure OpenAI / OpenAI"]:::external
        Speech["Azure Speech / Whisper"]:::external
        Storage["Azure Blob Storage"]:::external
    end

    %% Infrastructure
    subgraph "Infrastructure"
        Docker["Docker Containers"]:::external
        ARM["ARM Templates & CI"]:::external
    end

    %% Connections
    CLI -->|HTTP POST /ingest| IngestRouter
    WebUI -->|HTTP GET /query| QueryRouter

    IngestRouter -->|Publish Event| EH
    EH -->|Consume| IngestCons
    IngestCons -->|Run| IngestPipeline
    IngestPipeline -->|Store Media| Storage
    IngestPipeline -->|Index Chunks| SearchDB
    IngestPipeline -->|Transcribe| Speech

    QueryRouter -->|call| ImageAgent
    QueryRouter -->|call| VideoAgent
    QueryRouter -->|optional| MCP

    ImageAgent -->|via| ProviderFactory
    VideoAgent -->|via| ProviderFactory

    ProviderFactory -->|LLM Calls| LLM
    ProviderFactory -->|Search Queries| SearchDB
    ProviderFactory -->|Storage Access| Storage
    ProviderFactory -->|Vision Services| LLM
    ProviderFactory -->|Speech| Speech

    Docker --> API
    Docker --> IngestCons
    Docker --> MCP
    ARM -->|Deploy| Docker

    %% Click Events
    click API "https://github.com/microsoft/mmctagent/blob/main/app/main.py"
    click IngestRouter "https://github.com/microsoft/mmctagent/blob/main/app/routers/ingestion.py"
    click QueryRouter "https://github.com/microsoft/mmctagent/blob/main/app/routers/query.py"
    click IngestRouter "https://github.com/microsoft/mmctagent/blob/main/app/services/ingestion_services.py"
    click QueryRouter "https://github.com/microsoft/mmctagent/blob/main/app/services/query_services.py"
    click IngestCons "https://github.com/microsoft/mmctagent/blob/main/app/ingestion_consumer.py"
    click ImageAgent "https://github.com/microsoft/mmctagent/blob/main/mmct/image_pipeline/agents/image_agent.py"
    click VideoAgent "https://github.com/microsoft/mmctagent/blob/main/mmct/video_pipeline/agents/video_agent.py"
    click IngestPipeline "https://github.com/microsoft/mmctagent/blob/main/mmct/video_pipeline/core/ingestion/ingestion_pipeline.py"
    click ProviderFactory "https://github.com/microsoft/mmctagent/blob/main/mmct/providers/factory.py"
    click MCP "https://github.com/microsoft/mmctagent/blob/main/mcp_server/main.py"
    click ARM "https://github.com/microsoft/mmctagent/tree/main/infra/arm_templates/"

    %% Styles
    classDef internal fill:#cfe2f3,stroke:#0366d6,stroke-width:1px;
    classDef core fill:#d9ead3,stroke:#2f6627,stroke-width:1px;
    classDef external fill:#fce5cd,stroke:#97310e,stroke-width:1px;
```
