# crate/mlflow-cratedb architecture

> Architecture diagram of the GitHub repository crate/mlflow-cratedb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crate/mlflow-cratedb
- Repository: https://github.com/crate/mlflow-cratedb

## Overview

1. **What this repository is**
   - This is a Python package, `mlflow-cratedb`, that adapts **MLflow Tracking** so MLflow can use **CrateDB** as its backend storage.
   - The repo is not a full MLflow fork; it is a thin compatibility layer built mostly from targeted patches and startup/bootstrap code.

2. **Main architectural boundary**
   - The important boundary is between **MLflow’s expected storage/database behavior** and **CrateDB’s SQL/database behavior**.
   - The package sits in the middle and translates or patches MLflow internals so tracking metadata, artifacts of metadata, model registry data, and job-related records can be stored and queried against CrateDB.

3. **Package entrypoints and runtime control**
   - `mlflow_cratedb/boot.py`, `cli.py`, and `server.py` are the main operational entrypoints.
   - These likely handle environment setup, patch activation, version/diagnostic commands, and launching MLflow-related server processes with the CrateDB adapter in place.

4. **Database adapter subsystem**
   - `mlflow_cratedb/adapter/` contains the CrateDB-specific schema and setup logic.
   - `adapter/ddl/cratedb.sql` and `adapter/ddl/drop.sql` define the database lifecycle for the adapter: create the required tables/schema and tear them down.
   - `adapter/setup_db.py` is the orchestration layer that applies those DDL statements to prepare CrateDB for MLflow usage.

5. **Monkey-patch layer for MLflow**
   - `mlflow_cratedb/patch/mlflow/` is the core compatibility layer.
   - Files like `db_types.py`, `db_utils.py`, `tracking.py`, `server.py`, `settings.py`, `model.py`, and `search_utils.py` indicate the adapter patches MLflow’s database types, persistence helpers, tracking APIs, server behavior, configuration handling, model-related logic, and search/query behavior.
   - This is the main place where MLflow internals are altered rather than replaced.

6. **Job execution and background processing**
   - `mlflow_cratedb/patch/mlflow/jobs/` patches MLflow’s job execution path.
   - The presence of `_job_runner.py`, `huey_consumer.py`, and `utils.py` suggests the adapter also adjusts how MLflow background jobs are queued/consumed/executed, likely to keep those workflows compatible with the CrateDB-backed setup.

7. **SQLAlchemy integration**
   - `mlflow_cratedb/patch/sqlalchemy.py` indicates there is a compatibility shim at the ORM/DB-driver layer.
   - This is important because MLflow’s persistence layer typically sits on top of SQLAlchemy, so the adapter needs to reconcile SQLAlchemy-driven behavior with CrateDB specifics.

8. **External integration with PyCaret**
   - `mlflow_cratedb/patch/pycaret/mlflow_logger.py` and the `examples/tracking_pycaret.py` example show the package also supports PyCaret’s MLflow logging path.
   - This means the repository is not only about raw MLflow internals, but also about keeping third-party ML tooling working when MLflow is pointed at CrateDB.

9. **Data flow at a high level**
   - A user runs MLflow tooling, examples, or a server/CLI entrypoint.
   - Bootstrapping activates the MLflow patches.
   - MLflow operations that would normally target another backend are redirected through the adapter and patched persistence/search/job layers.
   - The adapter applies CrateDB-specific schema and SQL behavior, and MLflow tracking data is then stored and queried in CrateDB.

10. **Examples and documentation as usage boundaries**
   - `examples/` contains runnable tracking scenarios for dummy code, Merlion, and PyCaret, showing how the adapter is intended to be used from application code.
   - The `docs/` directory and README point to separate guides for handbook usage, container usage, and development workflows, suggesting the repository supports both library use and packaged deployment.

11. **Container and release packaging**
   - `release/compose/`, `release/oci-runtime/`, and `release/oci-server/` define containerized runtime targets.
   - This indicates the adapter can be delivered as container images in at least two modes: a runtime image and a server-oriented image, with self-tests for each.
   - The architecture therefore includes both library packaging and deployable container artifacts.

12. **Quality and validation**
   - `tests/` covers the adapter, CLI, MLflow patching, tracking behavior, job behavior, examples, and regression cases.
   - This test layout mirrors the architecture: it validates both the compatibility layer and the end-to-end behavior of MLflow talking to CrateDB.

13. **Overall system shape**
   - Conceptually, the repository is a **compatibility adapter layer** rather than a standalone service.
   - Its core subsystems are: initialization/CLI, CrateDB schema management, MLflow patching, job/runtime compatibility, optional third-party integrations, and deployment packaging.
   - For an architecture diagram, the central flow is: **application or MLflow server → adapter bootstrap/patches → CrateDB-backed persistence and query layer**.

Main entry points: [`mlflow_cratedb/patch/mlflow/tracking.py`](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/tracking.py) (Tracking), [`mlflow_cratedb/boot.py`](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/boot.py) (Boot), [`mlflow_cratedb/adapter/setup_db.py`](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/setup_db.py) (DB setup), [`mlflow_cratedb/server.py`](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/server.py) (Server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entrypoints"]
  node_mlflow_cratedb_boot["Boot<br/>[boot.py]"]
  node_mlflow_cratedb_cli["CLI<br/>command<br/>[cli.py]"]
  node_mlflow_cratedb_server["Server<br/>launcher<br/>[server.py]"]
end

subgraph group_adapter["CrateDB Adapter"]
  node_adapter_setup_db["DB setup<br/>orchestrator<br/>[setup_db.py]"]
  node_adapter_cratedb_sql["Create schema<br/>ddl<br/>[cratedb.sql]"]
  node_adapter_drop_sql["Drop schema<br/>ddl<br/>[drop.sql]"]
end

subgraph group_compat["MLflow Patches"]
  node_patch_tracking["Tracking<br/>patch<br/>[tracking.py]"]
  node_patch_db_utils["DB utils<br/>patch<br/>[db_utils.py]"]
  node_patch_db_types["DB types<br/>patch<br/>[db_types.py]"]
  node_patch_server["MLflow server<br/>patch<br/>[server.py]"]
  node_patch_settings["Settings<br/>patch<br/>[settings.py]"]
  node_patch_model["Model<br/>patch<br/>[model.py]"]
  node_patch_search_utils["Search<br/>patch<br/>[search_utils.py]"]
  node_patch_jobs_runner["Job runner<br/>jobs patch<br/>[_job_runner.py]"]
  node_patch_jobs_consumer["Huey consumer<br/>jobs patch<br/>[huey_consumer.py]"]
  node_patch_sqlalchemy["SQLAlchemy shim<br/>compat shim<br/>[sqlalchemy.py]"]
end

subgraph group_integrations["External Integrations"]
  node_patch_pycaret_logger["PyCaret logger<br/>integration patch<br/>[mlflow_logger.py]"]
  node_example_pycaret["PyCaret example"]
  node_example_dummy["Tracking example<br/>[tracking_dummy.py]"]
end

subgraph group_delivery["Delivery"]
  node_release_compose["Compose<br/>deployment<br/>[compose.yml]"]
  node_oci_runtime["OCI runtime<br/>image"]
  node_oci_server["OCI server<br/>image"]
end

node_mlflow_cratedb_cli -->|"starts"| node_mlflow_cratedb_boot
node_mlflow_cratedb_server -->|"starts"| node_mlflow_cratedb_boot
node_mlflow_cratedb_boot -->|"installs"| node_patch_sqlalchemy
node_mlflow_cratedb_boot -->|"activates"| node_patch_tracking
node_mlflow_cratedb_boot -->|"activates"| node_patch_server
node_mlflow_cratedb_boot -->|"prepares"| node_adapter_setup_db
node_adapter_setup_db -->|"applies"| node_adapter_cratedb_sql
node_adapter_setup_db -->|"reverses"| node_adapter_drop_sql
node_patch_tracking -->|"uses"| node_patch_db_utils
node_patch_tracking -->|"uses"| node_patch_db_types
node_patch_tracking -->|"uses"| node_patch_search_utils
node_patch_tracking -->|"uses"| node_patch_settings
node_patch_tracking -->|"uses"| node_patch_model
node_patch_tracking -->|"depends on"| node_patch_sqlalchemy
node_patch_server -->|"coordinates"| node_patch_tracking
node_patch_jobs_runner -->|"coordinates"| node_patch_jobs_consumer
node_patch_jobs_runner -->|"depends on"| node_patch_sqlalchemy
node_patch_pycaret_logger -->|"logs through"| node_patch_tracking
node_example_pycaret -->|"demonstrates"| node_patch_pycaret_logger
node_example_dummy -->|"demonstrates"| node_patch_tracking
node_release_compose -->|"packages"| node_mlflow_cratedb_server
node_release_compose -->|"packages"| node_mlflow_cratedb_boot
node_oci_runtime -->|"builds"| node_mlflow_cratedb_boot
node_oci_server -->|"builds"| node_mlflow_cratedb_server

click node_mlflow_cratedb_boot "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/boot.py"
click node_mlflow_cratedb_cli "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/cli.py"
click node_mlflow_cratedb_server "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/server.py"
click node_adapter_setup_db "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/setup_db.py"
click node_adapter_cratedb_sql "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/ddl/cratedb.sql"
click node_adapter_drop_sql "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/ddl/drop.sql"
click node_patch_tracking "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/tracking.py"
click node_patch_db_utils "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/db_utils.py"
click node_patch_db_types "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/db_types.py"
click node_patch_server "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/server.py"
click node_patch_settings "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/settings.py"
click node_patch_model "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/model.py"
click node_patch_search_utils "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/search_utils.py"
click node_patch_jobs_runner "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/jobs/_job_runner.py"
click node_patch_jobs_consumer "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/jobs/huey_consumer.py"
click node_patch_sqlalchemy "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/sqlalchemy.py"
click node_patch_pycaret_logger "https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/pycaret/mlflow_logger.py"
click node_example_pycaret "https://github.com/crate/mlflow-cratedb/blob/main/examples/tracking_pycaret.py"
click node_example_dummy "https://github.com/crate/mlflow-cratedb/blob/main/examples/tracking_dummy.py"
click node_release_compose "https://github.com/crate/mlflow-cratedb/blob/main/release/compose/compose.yml"
click node_oci_runtime "https://github.com/crate/mlflow-cratedb/tree/main/release/oci-runtime/Dockerfile"
click node_oci_server "https://github.com/crate/mlflow-cratedb/tree/main/release/oci-server/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mlflow_cratedb_boot,node_mlflow_cratedb_cli,node_mlflow_cratedb_server toneBlue
class node_adapter_setup_db,node_adapter_cratedb_sql,node_adapter_drop_sql toneAmber
class node_patch_tracking,node_patch_db_utils,node_patch_db_types,node_patch_server,node_patch_settings,node_patch_model,node_patch_search_utils,node_patch_jobs_runner,node_patch_jobs_consumer,node_patch_sqlalchemy toneMint
class node_patch_pycaret_logger,node_example_pycaret,node_example_dummy toneRose
class node_release_compose,node_oci_runtime,node_oci_server toneIndigo
```

## Components

### Entrypoints

Startup and command-line surfaces that activate the adapter.

- [Boot](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/boot.py): bootstrap · Initializes runtime state and activates adapter patches. · `mlflow_cratedb/boot.py`
- [CLI](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/cli.py): command · User-facing command surface for setup and diagnostics. · `mlflow_cratedb/cli.py`
- [Server](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/server.py): launcher · Starts MLflow-oriented server processes with CrateDB support. · `mlflow_cratedb/server.py`

### CrateDB Adapter

Schema management and database lifecycle for CrateDB-backed MLflow storage.

- [DB setup](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/setup_db.py): orchestrator · Applies CrateDB schema and teardown logic for the adapter. · `mlflow_cratedb/adapter/setup_db.py`
- [Create schema](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/ddl/cratedb.sql): ddl · CrateDB schema and table definitions for MLflow metadata. · `mlflow_cratedb/adapter/ddl/cratedb.sql`
- [Drop schema](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/adapter/ddl/drop.sql): ddl · Teardown SQL for removing adapter-managed database objects. · `mlflow_cratedb/adapter/ddl/drop.sql`

### MLflow Patches

Monkey patches that adapt MLflow internals to CrateDB behavior.

- [Tracking](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/tracking.py): patch · Redirects MLflow tracking persistence to CrateDB-compatible behavior. · `mlflow_cratedb/patch/mlflow/tracking.py`
- [DB utils](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/db_utils.py): patch · Patches MLflow database helper behavior and persistence helpers. · `mlflow_cratedb/patch/mlflow/db_utils.py`
- [DB types](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/db_types.py): patch · Adjusts database type handling for CrateDB compatibility. · `mlflow_cratedb/patch/mlflow/db_types.py`
- [MLflow server](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/server.py): patch · Adapts MLflow server behavior when the adapter is active. · `mlflow_cratedb/patch/mlflow/server.py`
- [Settings](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/settings.py): patch · Overrides or harmonizes MLflow configuration handling. · `mlflow_cratedb/patch/mlflow/settings.py`
- [Model](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/model.py): patch · Adjusts model registry or model-related persistence behavior. · `mlflow_cratedb/patch/mlflow/model.py`
- [Search](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/search_utils.py): patch · Patches query and search behavior for MLflow lookups. · `mlflow_cratedb/patch/mlflow/search_utils.py`
- [Job runner](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/jobs/_job_runner.py): jobs patch · Compatibility layer for MLflow job execution. · `mlflow_cratedb/patch/mlflow/jobs/_job_runner.py`
- [Huey consumer](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/mlflow/jobs/huey_consumer.py): jobs patch · Background job consumer integration for patched MLflow jobs. · `mlflow_cratedb/patch/mlflow/jobs/huey_consumer.py`
- [SQLAlchemy shim](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/sqlalchemy.py): compat shim · Reconciles SQLAlchemy behavior with CrateDB and MLflow expectations. · `mlflow_cratedb/patch/sqlalchemy.py`

### External Integrations

Third-party logging and example application paths that exercise the adapter.

- [PyCaret logger](https://github.com/crate/mlflow-cratedb/blob/main/mlflow_cratedb/patch/pycaret/mlflow_logger.py): integration patch · Routes PyCaret MLflow logging through the compatibility layer. · `mlflow_cratedb/patch/pycaret/mlflow_logger.py`
- [PyCaret example](https://github.com/crate/mlflow-cratedb/blob/main/examples/tracking_pycaret.py): example · Runnable example showing PyCaret tracking with CrateDB-backed MLflow. · `examples/tracking_pycaret.py`
- [Tracking example](https://github.com/crate/mlflow-cratedb/blob/main/examples/tracking_dummy.py): example · Basic tracking scenario used to demonstrate adapter usage. · `examples/tracking_dummy.py`

### Delivery

Container and release artifacts for packaged runtime and server modes.

- [Compose](https://github.com/crate/mlflow-cratedb/blob/main/release/compose/compose.yml): deployment · Container orchestration for packaged adapter/runtime usage. · `release/compose/compose.yml`
- [OCI runtime](https://github.com/crate/mlflow-cratedb/tree/main/release/oci-runtime/Dockerfile): image · Runtime container image for adapter-enabled execution. · `release/oci-runtime/Dockerfile`
- [OCI server](https://github.com/crate/mlflow-cratedb/tree/main/release/oci-server/Dockerfile): image · Server-oriented container image for running the adapter with MLflow. · `release/oci-server/Dockerfile`

## Connections

- CLI → Boot: starts · CLI invokes bootstrap before doing work.
- Server → Boot: starts · Server entrypoint activates bootstrap and patches.
- Boot → SQLAlchemy shim: installs · Bootstrap prepares compatibility shims.
- Boot → Tracking: activates · Bootstrap enables the MLflow tracking patch set.
- Boot → MLflow server: activates · Bootstrap enables server-side MLflow adjustments.
- Boot → DB setup: prepares · Bootstrap may initialize the CrateDB schema.
- DB setup → Create schema: applies · Setup uses create-schema DDL.
- DB setup → Drop schema: reverses · Setup uses drop-schema DDL for teardown.
- Tracking → DB utils: uses · Tracking patch relies on DB helper overrides.
- Tracking → DB types: uses · Tracking patch relies on type compatibility changes.
- Tracking → Search: uses · Tracking patch relies on patched query behavior.
- Tracking → Settings: uses · Tracking patch consults adjusted settings behavior.
- Tracking → Model: uses · Tracking patch shares model-related compatibility hooks.
- Tracking → SQLAlchemy shim: depends on · Tracking persistence sits on the SQLAlchemy shim.
- MLflow server → Tracking: coordinates · Server behavior is aligned with tracking persistence patches.
- Job runner → Huey consumer: coordinates · Job runner and consumer form the patched background execution path.
- Job runner → SQLAlchemy shim: depends on · Jobs path uses database compatibility support.
- PyCaret logger → Tracking: logs through · PyCaret integration emits MLflow tracking calls through the patch layer.
- PyCaret example → PyCaret logger: demonstrates · Example shows third-party logging integration.
- Tracking example → Tracking: demonstrates · Example exercises the tracking compatibility layer.
- Compose → Server: packages · Compose deployment targets the server entrypoint.
- Compose → Boot: packages · Compose deployment relies on bootstrap initialization.
- OCI runtime → Boot: builds · Runtime image includes bootstrap for adapter activation.
- OCI server → Server: builds · Server image wraps the server launcher.
