How microsoft/mmctagent works
Project Type and Purpose
• MMCTAgent is a modular, full-stack AI framework for multi-modal critical thinking and complex visual reasoning.
• It provides:
• ImageAgent (static image Q&A with planner/critic loop)
• VideoAgent (video ingestion, indexing, and Q&A pipelines)
• A FastAPI front-end (‘app’) exposing ingestion and query endpoints
• An optional MCP server microservice for orchestrating planner/critic loops
• It is research-oriented, vendor-agnostic (pluggable provider system), and deployable to Azure via Docker/ARM templates.
File-Structure Analysis
Top-level directories:
── infra/ (ARM templates + Bash scripts for Azure infra: Event Hub, AI Search, OpenAI, Blob Storage, App Services)
── app/ (FastAPI microservice: ingestion_consumer, main API, routers, schemas, services, utilities)
── mcp_server/ (standalone TCP/HTTP server + client for planner/critic orchestration)
── mmct/ (core Python library)
• image_pipeline/ (ImageAgent entrypoint + tools/models)
• video_pipeline/ (IngestionPipeline + VideoAgent + keyframes, chapter generation, transcription)
• providers/ (factory + base + azure/openai/custom adapters for LLM, search, storage, vision, speech)
• config/, security/, utils/, tests/
── examples/, README files, Dockerfiles, CI support files.
Key Insights from README
• Two specialized agents (ImageAgent & VideoAgent) implementing “planner → critic → refine” loops
• Multi-cloud provider abstraction (Azure + OpenAI + local FAISS)
• FASTAPI “app” runs on Docker containers, connects to Azure Event Hub for ingestion, Azure AI Search / FAISS for vector DB, OpenAI/Azure OpenAI for LLM, Blob Storage for assets.
• infra/ contains deployment scripts and ARM templates for all needed Azure services.