# siddharthiitian/mindsdb architecture

> Architecture diagram of the GitHub repository siddharthiitian/mindsdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siddharthiitian/mindsdb
- Repository: https://github.com/siddharthiitian/mindsdb

## Overview

##### Project type and purpose
- This is an open-source, full-stack “AI Virtual Database” platform (MindsDB) written in Python.
- It exposes data-source–agnostic AI/ML model training and inference via:
- SQL proxies (MySQL, PostgreSQL, MongoDB)
- HTTP/REST API
- gRPC microservices
- It dynamically loads “handlers” for 130+ data integrations (relational, NoSQL, SaaS) and AI engines (OpenAI, HuggingFace, custom models).

##### File structure
- Top-level directories:
- mindsdb/ – core Python code: API layer, proxies, microservices, integrations, interfaces, utilities
- docker/ & docker-compose.yml – multi-container deployment (API, executor, NGINX, handler discovery, databases)
- helm/ – Kubernetes deployment templates
- protos/ – Protocol Buffer definitions for gRPC services
- migrations/ – Alembic DB migrations for metadata storage
- tests/ – unit, handler tests, integration tests
- Key architectural modules:
- api/http – Flask/Gunicorn HTTP server & endpoint namespaces
- api/*/proxy – Binary SQL protocol proxies (MySQL, PostgreSQL, Mongo)
- microservices_grpc – gRPC stubs for “db” and “ml” services
- integrations/handlers – plugin directory, each handler implements data-source adapter or AI engine adapter
- utilities/ml_task_queue – task queue for asynchronous ML jobs
- interfaces/ – controllers abstracting high-level operations (jobs, models, storage)

##### README for additional insights
- Purpose: “Connect any AI/ML model to any data source”, automated training & inference pipelines
- How it works: Connect → Create Model → Query Models → Fine-Tune → Automate Workflows
- Supported access patterns: SQL SELECT, REST calls, JOINs, SDKs (Python/JavaScript)

##### Main components & interactions
a. Clients
- SDK / CLI / GUI / SQL clients →
b. API Gateway
- NGINX load-balances SQL ports + HTTP endpoints →
c. SQL Proxies
- MySQL Proxy, PostgreSQL Proxy, MongoDB Proxy
- Speak each protocol, forward queries to gRPC wrappers
d. HTTP API
- Flask/Gunicorn server hosting REST endpoints →
e. Handler Discovery Service
- Discovers & loads integration plugins at startup
f. gRPC Microservices
- db service: CRUD operations on metadata, projects, storage
- ml service: Dispatch training/inference tasks
g. Core Engine
- Task queue for ML jobs (producer/consumer)
- Scheduler, migration manager, telemetry, logging
h. Integration Handlers (plugins)
- Adapters to external data-sources & AI engines
i. External Systems
- Databases (MySQL, PostgreSQL, Mongo, others)
- AI engines (OpenAI, HuggingFace, custom BYOM)

##### Architectural patterns & design principles
- Microservices: distinct gRPC services for DB metadata vs. ML task execution
- Adapter/Plugin: one directory per integration, implements a common handler interface
- Proxy Pattern: front-end proxies implement native SQL protocols
- Task Queue: decouple request from background ML jobs via producer/consumer
- Infrastructure as Code: Docker Compose for local development, Helm for Kubernetes

##### Technologies & frameworks
- Python 3.8+
- Flask / Gunicorn (HTTP API)
- Protocol buffers & gRPC (inter-service comms)
- Docker / Docker Compose / Helm (deployment)
- Alembic (DB migrations)
- Pre-commit, GitHub Actions (CI/CD)

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Clients Layer
  subgraph "Clients" 
    direction TB
    Clients["SDK / CLI / GUI / SQL Clients"]
  end

  %% API Gateway Layer
  subgraph "API Gateway" 
    direction TB
    APIGateway["NGINX Load Balancer"]:::gateway
  end

  %% Proxies and HTTP API Layer
  subgraph "SQL Proxies & HTTP API"
    direction TB
    MySQLProxy["MySQL Proxy"]:::proxy
    PostgreProxy["PostgreSQL Proxy"]:::proxy
    MongoProxy["MongoDB Proxy"]:::proxy
    HTTPAPI["HTTP API (Flask/Gunicorn)"]:::api
  end

  %% Handler Discovery & gRPC Wrapper Layer
  subgraph "Handler Discovery & gRPC Wrapper"
    direction TB
    HandlerDiscovery["Handler Discovery Service"]:::service
    subgraph "gRPC Microservices"
      direction TB
      DBService["DB Service"]:::service
      MLService["ML Service"]:::service
      ExecutorBridge["Executor Bridge"]:::service
    end
  end

  %% Core Engine Layer
  subgraph "Core Engine"
    direction TB
    TaskQueue["ML Task Queue"]:::core
    Scheduler["Scheduler"]:::core
    Migrations["Migrations"]:::core
    Telemetry["Telemetry & Logging"]:::core
  end

  %% Integration Handlers Layer
  subgraph "Integration Handlers"
    direction TB
    Handlers["Plugins (mindsdb/integrations/handlers)"]:::plugin
  end

  %% External Systems Layer
  subgraph "External Systems"
    direction TB
    Databases["Databases (MySQL, PostgreSQL, Mongo, etc.)"]:::external
    AIEngines["AI Engines (OpenAI, HuggingFace, custom)"]:::external
  end

  %% Connections
  Clients -->|“requests”| APIGateway
  APIGateway -->|“MySQL wire / Postgre wire / Mongo wire”| MySQLProxy
  APIGateway -->|“MySQL wire / Postgre wire / Mongo wire”| PostgreProxy
  APIGateway -->|“MySQL wire / Postgre wire / Mongo wire”| MongoProxy
  APIGateway -->|“HTTP/REST”| HTTPAPI

  MySQLProxy -->|“gRPC”| DBService
  PostgreProxy -->|“gRPC”| DBService
  MongoProxy -->|“gRPC”| DBService
  HTTPAPI -->|“gRPC”| DBService
  HTTPAPI -->|“gRPC”| MLService

  HandlerDiscovery -->|“load plugins”| Handlers

  DBService -->|“metadata ops”| TaskQueue
  MLService -->|“dispatch jobs”| TaskQueue
  ExecutorBridge -->|“bridges tasks”| TaskQueue

  TaskQueue -->|“schedule/run”| Scheduler
  Scheduler --> Migrations
  Scheduler --> Telemetry

  TaskQueue -->|“invoke adapter”| Handlers
  Handlers -->|“query/write”| Databases
  Handlers -->|“call”| AIEngines

  Databases -->|“results”| DBService
  AIEngines -->|“responses”| MLService

  DBService -->|“results”| HTTPAPI
  MLService -->|“results”| HTTPAPI
  DBService -->|“results”| Clients
  MLService -->|“results”| Clients

  %% Click Events
  click APIGateway "https://github.com/siddharthiitian/mindsdb/blob/staging/docker/nginx/templates/default.conf.template"
  click APIGateway "https://github.com/siddharthiitian/mindsdb/blob/staging/docker/nginx-compose.yml"
  click MySQLProxy "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/api/mysql/mysql_proxy/"
  click PostgreProxy "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/api/postgres/postgres_proxy/"
  click MongoProxy "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/api/mongo/"
  click HTTPAPI "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/api/http/"
  click HTTPAPI "https://github.com/siddharthiitian/mindsdb/blob/staging/mindsdb/api/http/gunicorn_wrapper.py"
  click HTTPAPI "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/api/http/namespaces/"
  click HandlerDiscovery "https://github.com/siddharthiitian/mindsdb/blob/staging/docker/handler_discovery/sd.py"
  click HandlerDiscovery "https://github.com/siddharthiitian/mindsdb/blob/staging/mindsdb/integrations/handlers/__init__.py"
  click DBService "https://github.com/siddharthiitian/mindsdb/blob/staging/protos/db.proto"
  click DBService "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/microservices_grpc/db/"
  click MLService "https://github.com/siddharthiitian/mindsdb/blob/staging/protos/ml.proto"
  click MLService "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/microservices_grpc/ml/"
  click ExecutorBridge "https://github.com/siddharthiitian/mindsdb/blob/staging/protos/executor.proto"
  click ExecutorBridge "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/microservices_grpc/executor/"
  click TaskQueue "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/utilities/ml_task_queue/"
  click Scheduler "https://github.com/siddharthiitian/mindsdb/blob/staging/mindsdb/interfaces/jobs/scheduler.py"
  click Migrations "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/migrations/"
  click Telemetry "https://github.com/siddharthiitian/mindsdb/blob/staging/mindsdb/utilities/telemetry.py"
  click Telemetry "https://github.com/siddharthiitian/mindsdb/blob/staging/mindsdb/utilities/log_controller.py"
  click Handlers "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/integrations/handlers/"
  click Handlers "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/integrations/handlers/openai_handler/"
  click Handlers "https://github.com/siddharthiitian/mindsdb/tree/staging/mindsdb/integrations/handlers/mysql_handler/"
  click Databases "https://github.com/siddharthiitian/mindsdb/blob/staging/docker-compose.yml"
  click Databases "https://github.com/siddharthiitian/mindsdb/tree/staging/helm/mindsdb/"
  click Databases "https://github.com/siddharthiitian/mindsdb/blob/staging/docker/*.Dockerfile"

  %% Styles
  classDef external fill:#D6EAF8,stroke:#1B4F72
  classDef service fill:#D5F5E3,stroke:#145A32
  classDef plugin fill:#E5E7E9,stroke:#626567
  classDef gateway fill:#FADBD8,stroke:#943126
  classDef proxy fill:#FCF3CF,stroke:#7D6608
  classDef core fill:#FDEDEC,stroke:#641E16
  classDef api fill:#EBF5FB,stroke:#1A5276
```
