How siddharthiitian/mindsdb works
Project type and purpose
• This is an open-source, full-stack “AI Virtual Database” platform (MindsDB) written in Python.
• It exposes data-source–agnostic AI/ML model training and inference via:
• SQL proxies (MySQL, PostgreSQL, MongoDB)
• HTTP/REST API
• gRPC microservices
• It dynamically loads “handlers” for 130+ data integrations (relational, NoSQL, SaaS) and AI engines (OpenAI, HuggingFace, custom models).
File structure
• Top-level directories:
• mindsdb/ – core Python code: API layer, proxies, microservices, integrations, interfaces, utilities
• docker/ & docker-compose.yml – multi-container deployment (API, executor, NGINX, handler discovery, databases)
• helm/ – Kubernetes deployment templates
• protos/ – Protocol Buffer definitions for gRPC services
• migrations/ – Alembic DB migrations for metadata storage
• tests/ – unit, handler tests, integration tests
• Key architectural modules:
• api/http – Flask/Gunicorn HTTP server & endpoint namespaces
• api/*/proxy – Binary SQL protocol proxies (MySQL, PostgreSQL, Mongo)
• microservices_grpc – gRPC stubs for “db” and “ml” services
• integrations/handlers – plugin directory, each handler implements data-source adapter or AI engine adapter
• utilities/ml_task_queue – task queue for asynchronous ML jobs
• interfaces/ – controllers abstracting high-level operations (jobs, models, storage)
README for additional insights
• Purpose: “Connect any AI/ML model to any data source”, automated training & inference pipelines
• How it works: Connect → Create Model → Query Models → Fine-Tune → Automate Workflows
• Supported access patterns: SQL SELECT, REST calls, JOINs, SDKs (Python/JavaScript)
Main components & interactions
a. Clients
• SDK / CLI / GUI / SQL clients →
b. API Gateway
• NGINX load-balances SQL ports + HTTP endpoints →
c. SQL Proxies
• MySQL Proxy, PostgreSQL Proxy, MongoDB Proxy
• Speak each protocol, forward queries to gRPC wrappers
d. HTTP API
• Flask/Gunicorn server hosting REST endpoints →
e. Handler Discovery Service
• Discovers & loads integration plugins at startup
f. gRPC Microservices
• db service: CRUD operations on metadata, projects, storage
• ml service: Dispatch training/inference tasks
g. Core Engine
• Task queue for ML jobs (producer/consumer)
• Scheduler, migration manager, telemetry, logging
h. Integration Handlers (plugins)
• Adapters to external data-sources & AI engines
i. External Systems
• Databases (MySQL, PostgreSQL, Mongo, others)
• AI engines (OpenAI, HuggingFace, custom BYOM)
Architectural patterns & design principles
• Microservices: distinct gRPC services for DB metadata vs. ML task execution
• Adapter/Plugin: one directory per integration, implements a common handler interface
• Proxy Pattern: front-end proxies implement native SQL protocols
• Task Queue: decouple request from background ML jobs via producer/consumer
• Infrastructure as Code: Docker Compose for local development, Helm for Kubernetes
Technologies & frameworks
• Python 3.8+
• Flask / Gunicorn (HTTP API)
• Protocol buffers & gRPC (inter-service comms)
• Docker / Docker Compose / Helm (deployment)
• Alembic (DB migrations)
• Pre-commit, GitHub Actions (CI/CD)