# multi-modal-ai/multimodal-agents-course architecture

> Architecture diagram of the GitHub repository multi-modal-ai/multimodal-agents-course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/multi-modal-ai/multimodal-agents-course
- Repository: https://github.com/multi-modal-ai/multimodal-agents-course

## Overview

##### Project Type & Purpose
- This is a **full-stack, microservices-style AI agent application** built as an educational course.
- Purpose:
- Teach how to build a multimodal processing pipeline (video, image, audio, text) called the **MCP server**.
- How to build an **Agent API** that uses MCP Tools and LLMs (Groq, OpenAI).
- Provide a **React UI** for interactive agent conversations with video-powered intelligence.

##### File Structure
- Top-level microservices:
- **kubrick-mcp** – Python FastMCP server + video ingestion pipeline (src/kubrick_mcp/video/…)
- **kubrick-api** – Python FastAPI Agent server and custom MCP client + memory layer (Pixeltable)
- **kubrick-ui** – React + Vite frontend served via Nginx
- Shared config: Dockerfiles, docker-compose.yml for local orchestration, Makefiles, .env files.
- Notebooks for playground/experimentation in each service.

##### Key README Insights
- The system uses **FastMCP** for protocol transport between Agent and MCP server.
- **Pixeltable** provides stateful memory for agents and video data indexing.
- **Opik** (Comet) is integrated for prompt versioning, tracing, observability.
- External AI services: **Groq** (VLM & custom tools) and **OpenAI** (LLM & embeddings).

##### Main Components & Relationships
A. Frontend (kubrick-ui)
- React UI (Chat interface, Video sidebar)
- Served via Nginx in Docker
- Communicates over HTTP/WebSocket to Agent API
B. Agent API (kubrick-api)
- FastAPI server exposing `/chat`, `/history`, `/video-search` endpoints
- MCP client (FastMCP) calls into MCP server tools
- Agent logic:
- BaseAgent, GroqAgent, memory (Pixeltable)
- Tools wrapper (groq_tool.py, kubrick_api/tools.py)
- Opik integration for tracing and prompt versioning
C. MCP Server (kubrick-mcp)
- FastMCP server exposing Tools, Resources, Prompts
- Video ingestion pipeline modules:
- ingestion/functions, registry, video_processor
- video_search_engine for embeddings & search
- Uses Pixeltable to store multimodal data & embeddings
D. External Services
- Groq API for Vision & Language models
- OpenAI API for LLM calls & embeddings
- Comet Opik for observability

##### Architectural Patterns & Principles
- **Microservices**: separate UI, API, MCP server
- **Client-Server**: React client → FastAPI agent server → FastMCP MCP server
- **Pipeline Pattern**: video ingestion → processing → embedding → search
- **Tool-Use Agent**: abstract tools via MCP
- **Observability**: integrate Opik for tracing across services

##### Elements
- Clear labels on all components (e.g., “kubrick-ui (React + Nginx)”)
- Container boundaries to show Docker/docker-compose or Kubernetes pods

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Service
    subgraph "Frontend (kubrick-ui)"
        direction TB
        UI_Dir["kubrick-ui/"]:::frontend
        Dockerfile_UI["Dockerfile"]:::frontend
        NginxConf["nginx.conf"]:::frontend
        ReactEntry["src/main.tsx"]:::frontend
        ComponentsDir["src/components/"]:::frontend
        PagesDir["src/pages/"]:::frontend
        PublicDir["public/"]:::frontend
    end

    %% Agent API Service
    subgraph "Agent API (kubrick-api)"
        direction TB
        API_Dir["kubrick-api/"]:::api
        Entry["src/kubrick_api/api.py"]:::api
        ConfigAPI["src/kubrick_api/config.py"]:::api
        ModelsAPI["src/kubrick_api/models.py"]:::api
        OpikUtils["src/kubrick_api/opik_utils.py"]:::api
        ToolsAPI["src/kubrick_api/tools.py"]:::api
        BaseAgent["agent/base_agent.py"]:::api
        GroqAgent["agent/groq/groq_agent.py"]:::api
        GroqTool["agent/groq/groq_tool.py"]:::api
        Memory["agent/memory.py"]:::api
        NotebooksAPI["notebooks/"]:::api
    end

    %% MCP Server Service
    subgraph "MCP Server (kubrick-mcp)"
        direction TB
        MCP_Dir["kubrick-mcp/"]:::mcp
        Server["src/kubrick_mcp/server.py"]:::mcp
        ConfigMCP["src/kubrick_mcp/config.py"]:::mcp
        Prompts["src/kubrick_mcp/prompts.py"]:::mcp
        Resources["src/kubrick_mcp/resources.py"]:::mcp
        ToolsMCP["src/kubrick_mcp/tools.py"]:::mcp
        subgraph "Video Ingestion Pipeline"
            direction TB
            IngestFuncs["video/ingestion/functions.py"]:::mcp
            IngestConsts["video/ingestion/constants.py"]:::mcp
            IngestModels["video/ingestion/models.py"]:::mcp
            IngestRegistry["video/ingestion/registry.py"]:::mcp
            VideoProcessor["video/ingestion/video_processor.py"]:::mcp
        end
        SearchEngine["video/video_search_engine.py"]:::mcp
        NotebooksMCP["notebooks/"]:::mcp
    end

    %% Shared Configuration & Orchestration
    subgraph "Shared Config & Orchestration"
        direction TB
        DC["docker-compose.yml / docker-compose.dev.yml"]:::shared
        Makefile["Makefile"]:::shared
        EnvRoot[".env.example"]:::shared
        PythonRoot[".python-version"]:::shared
        Dockerignore[".dockerignore"]:::shared
        EnvAPI["kubrick-api/.env.example"]:::shared
        PythonAPI["kubrick-api/.python-version"]:::shared
        EnvMCP["kubrick-mcp/.env.example"]:::shared
        PythonMCP["kubrick-mcp/.python-version"]:::shared
    end

    %% Data Stores
    PixeltableDB[("Pixeltable DB")]:::database

    %% External Services
    subgraph "External Services"
        direction TB
        GroqAPI(["Groq API"]):::external
        OpenAIAPI(["OpenAI API"]):::external
        OpikService(["Comet Opik"]):::external
    end

    %% Connections
    UI_Dir -->|"HTTP/WebSocket"| Entry
    Entry -->|"FastMCP"| Server
    Server --> VideoProcessor
    VideoProcessor --> SearchEngine
    SearchEngine --> PixeltableDB
    Server --> PixeltableDB
    Entry -->|"LLM & Embeddings"| OpenAIAPI
    Entry -->|"Vision & Language"| GroqAPI
    Server -->|"VLM & Tools"| GroqAPI
    Server -->|"Embeddings"| OpenAIAPI
    Entry -.->|"Tracing"| OpikService
    Server -.->|"Tracing"| OpikService

    %% Click Events Frontend
    click UI_Dir "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-ui/"
    click Dockerfile_UI "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-ui/Dockerfile"
    click NginxConf "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-ui/nginx.conf"
    click ReactEntry "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-ui/src/main.tsx"
    click ComponentsDir "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-ui/src/components/"
    click PagesDir "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-ui/src/pages/"
    click PublicDir "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-ui/public/"

    %% Click Events Agent API
    click API_Dir "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-api/"
    click Entry "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/api.py"
    click ConfigAPI "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/config.py"
    click ModelsAPI "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/models.py"
    click OpikUtils "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/opik_utils.py"
    click ToolsAPI "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/tools.py"
    click BaseAgent "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/agent/base_agent.py"
    click GroqAgent "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/agent/groq/groq_agent.py"
    click GroqTool "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/agent/groq/groq_tool.py"
    click Memory "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/src/kubrick_api/agent/memory.py"
    click NotebooksAPI "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-api/notebooks/"

    %% Click Events MCP Server
    click MCP_Dir "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-mcp/"
    click Server "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/server.py"
    click ConfigMCP "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/config.py"
    click Prompts "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/prompts.py"
    click Resources "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/resources.py"
    click ToolsMCP "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/tools.py"
    click IngestFuncs "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/video/ingestion/functions.py"
    click IngestConsts "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/video/ingestion/constants.py"
    click IngestModels "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/video/ingestion/models.py"
    click IngestRegistry "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/video/ingestion/registry.py"
    click VideoProcessor "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/video/ingestion/video_processor.py"
    click SearchEngine "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/src/kubrick_mcp/video/video_search_engine.py"
    click NotebooksMCP "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/kubrick-mcp/notebooks/"

    %% Click Events Shared Config
    click DC "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/docker-compose.yml"
    click Makefile "https://github.com/multi-modal-ai/multimodal-agents-course/tree/main/Makefile"
    click EnvRoot "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/.env.example"
    click PythonRoot "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/.python-version"
    click Dockerignore "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/.dockerignore"
    click EnvAPI "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/.env.example"
    click PythonAPI "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-api/.python-version"
    click EnvMCP "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/.env.example"
    click PythonMCP "https://github.com/multi-modal-ai/multimodal-agents-course/blob/main/kubrick-mcp/.python-version"

    %% Style Definitions
    classDef frontend fill:#cce5ff,stroke:#66b2ff,color:#003366;
    classDef api fill:#d4edda,stroke:#81c784,color:#1b5e20;
    classDef mcp fill:#ffe5b4,stroke:#ffb84d,color:#663c00;
    classDef shared fill:#f0f0f0,stroke:#cccccc,color:#333333;
    classDef database fill:#e0e0e0,stroke:#999999,color:#000000,shape:cylinder;
    classDef external fill:#e1bee7,stroke:#ba68c8,color:#4a148c,shape:cloud;
```
