How multi-modal-ai/multimodal-agents-course works
Project Type & Purpose
• This is a full-stack, microservices-style AI agent application built as an educational course.
• Purpose:
• Teach how to build a multimodal processing pipeline (video, image, audio, text) called the MCP server.
• How to build an Agent API that uses MCP Tools and LLMs (Groq, OpenAI).
• Provide a React UI for interactive agent conversations with video-powered intelligence.
File Structure
• Top-level microservices:
• kubrick-mcp – Python FastMCP server + video ingestion pipeline (src/kubrick_mcp/video/…)
• kubrick-api – Python FastAPI Agent server and custom MCP client + memory layer (Pixeltable)
• kubrick-ui – React + Vite frontend served via Nginx
• Shared config: Dockerfiles, docker-compose.yml for local orchestration, Makefiles, .env files.
• Notebooks for playground/experimentation in each service.
Key README Insights
• The system uses FastMCP for protocol transport between Agent and MCP server.
• Pixeltable provides stateful memory for agents and video data indexing.
• Opik (Comet) is integrated for prompt versioning, tracing, observability.
• External AI services: Groq (VLM & custom tools) and OpenAI (LLM & embeddings).
Main Components & Relationships
A. Frontend (kubrick-ui)
• React UI (Chat interface, Video sidebar)
• Served via Nginx in Docker
• Communicates over HTTP/WebSocket to Agent API
B. Agent API (kubrick-api)
• FastAPI server exposing /chat, /history, /video-search endpoints
• MCP client (FastMCP) calls into MCP server tools
• Agent logic:
• BaseAgent, GroqAgent, memory (Pixeltable)
• Tools wrapper (groq_tool.py, kubrick_api/tools.py)
• Opik integration for tracing and prompt versioning
C. MCP Server (kubrick-mcp)
• FastMCP server exposing Tools, Resources, Prompts
• Video ingestion pipeline modules:
• ingestion/functions, registry, video_processor
• video_search_engine for embeddings & search
• Uses Pixeltable to store multimodal data & embeddings
D. External Services
• Groq API for Vision & Language models
• OpenAI API for LLM calls & embeddings
• Comet Opik for observability
Architectural Patterns & Principles
• Microservices: separate UI, API, MCP server
• Client-Server: React client → FastAPI agent server → FastMCP MCP server
• Pipeline Pattern: video ingestion → processing → embedding → search
• Tool-Use Agent: abstract tools via MCP
• Observability: integrate Opik for tracing across services
Elements
• Clear labels on all components (e.g., “kubrick-ui (React + Nginx)”)
• Container boundaries to show Docker/docker-compose or Kubernetes pods