How crate/mlflow-cratedb works
1. What this repository is
- This is a Python package, mlflow-cratedb, that adapts MLflow Tracking so MLflow can use CrateDB as its backend storage.
- The repo is not a full MLflow fork; it is a thin compatibility layer built mostly from targeted patches and startup/bootstrap code.
2. Main architectural boundary
- The important boundary is between MLflow’s expected storage/database behavior and CrateDB’s SQL/database behavior.
- The package sits in the middle and translates or patches MLflow internals so tracking metadata, artifacts of metadata, model registry data, and job-related records can be stored and queried against CrateDB.
3. Package entrypoints and runtime control
- mlflow_cratedb/boot.py, cli.py, and server.py are the main operational entrypoints.
- These likely handle environment setup, patch activation, version/diagnostic commands, and launching MLflow-related server processes with the CrateDB adapter in place.
4. Database adapter subsystem
- mlflow_cratedb/adapter/ contains the CrateDB-specific schema and setup logic.
- adapter/ddl/cratedb.sql and adapter/ddl/drop.sql define the database lifecycle for the adapter: create the required tables/schema and tear them down.
- adapter/setup_db.py is the orchestration layer that applies those DDL statements to prepare CrateDB for MLflow usage.
5. Monkey-patch layer for MLflow
- mlflow_cratedb/patch/mlflow/ is the core compatibility layer.
- Files like db_types.py, db_utils.py, tracking.py, server.py, settings.py, model.py, and search_utils.py indicate the adapter patches MLflow’s database types, persistence helpers, tracking APIs, server behavior, configuration handling, model-related logic, and search/query behavior.
- This is the main place where MLflow internals are altered rather than replaced.
6. Job execution and background processing
- mlflow_cratedb/patch/mlflow/jobs/ patches MLflow’s job execution path.
- The presence of _job_runner.py, huey_consumer.py, and utils.py suggests the adapter also adjusts how MLflow background jobs are queued/consumed/executed, likely to keep those workflows compatible with the CrateDB-backed setup.
7. SQLAlchemy integration
- mlflow_cratedb/patch/sqlalchemy.py indicates there is a compatibility shim at the ORM/DB-driver layer.
- This is important because MLflow’s persistence layer typically sits on top of SQLAlchemy, so the adapter needs to reconcile SQLAlchemy-driven behavior with CrateDB specifics.
8. External integration with PyCaret
- mlflow_cratedb/patch/pycaret/mlflow_logger.py and the examples/tracking_pycaret.py example show the package also supports PyCaret’s MLflow logging path.
- This means the repository is not only about raw MLflow internals, but also about keeping third-party ML tooling working when MLflow is pointed at CrateDB.
9. Data flow at a high level
- A user runs MLflow tooling, examples, or a server/CLI entrypoint.
- Bootstrapping activates the MLflow patches.
- MLflow operations that would normally target another backend are redirected through the adapter and patched persistence/search/job layers.
- The adapter applies CrateDB-specific schema and SQL behavior, and MLflow tracking data is then stored and queried in CrateDB.
10. Examples and documentation as usage boundaries
- examples/ contains runnable tracking scenarios for dummy code, Merlion, and PyCaret, showing how the adapter is intended to be used from application code.
- The docs/ directory and README point to separate guides for handbook usage, container usage, and development workflows, suggesting the repository supports both library use and packaged deployment.
11. Container and release packaging
- release/compose/, release/oci-runtime/, and release/oci-server/ define containerized runtime targets.
- This indicates the adapter can be delivered as container images in at least two modes: a runtime image and a server-oriented image, with self-tests for each.
- The architecture therefore includes both library packaging and deployable container artifacts.
12. Quality and validation
- tests/ covers the adapter, CLI, MLflow patching, tracking behavior, job behavior, examples, and regression cases.
- This test layout mirrors the architecture: it validates both the compatibility layer and the end-to-end behavior of MLflow talking to CrateDB.
13. Overall system shape
- Conceptually, the repository is a compatibility adapter layer rather than a standalone service.
- Its core subsystems are: initialization/CLI, CrateDB schema management, MLflow patching, job/runtime compatibility, optional third-party integrations, and deployment packaging.
- For an architecture diagram, the central flow is: application or MLflow server → adapter bootstrap/patches → CrateDB-backed persistence and query layer.
Main entry points: tracking.py Tracking · boot.py Boot · setup_db.py DB setup · server.py Server
Components 22
Entrypoints
Startup and command-line surfaces that activate the adapter.
CrateDB Adapter
Schema management and database lifecycle for CrateDB-backed MLflow storage.
- DB setup · Applies CrateDB schema and teardown logic for the adapter.
mlflow_cratedb/adapter/setup_db.py - Create schema · CrateDB schema and table definitions for MLflow metadata.
mlflow_cratedb/adapter/ddl/cratedb.sql - Drop schema · Teardown SQL for removing adapter-managed database objects.
mlflow_cratedb/adapter/ddl/drop.sql
MLflow Patches
Monkey patches that adapt MLflow internals to CrateDB behavior.
- Tracking · Redirects MLflow tracking persistence to CrateDB-compatible behavior.
mlflow_cratedb/patch/mlflow/tracking.py - DB utils · Patches MLflow database helper behavior and persistence helpers.
mlflow_cratedb/patch/mlflow/db_utils.py - DB types · Adjusts database type handling for CrateDB compatibility.
mlflow_cratedb/patch/mlflow/db_types.py - MLflow server · Adapts MLflow server behavior when the adapter is active.
mlflow_cratedb/patch/mlflow/server.py - Settings · Overrides or harmonizes MLflow configuration handling.
mlflow_cratedb/patch/mlflow/settings.py - Model · Adjusts model registry or model-related persistence behavior.
mlflow_cratedb/patch/mlflow/model.py - Search · Patches query and search behavior for MLflow lookups.
mlflow_cratedb/patch/mlflow/search_utils.py - Job runner · Compatibility layer for MLflow job execution.
mlflow_cratedb/patch/mlflow/jobs/_job_runner.py - Huey consumer · Background job consumer integration for patched MLflow jobs.
mlflow_cratedb/patch/mlflow/jobs/huey_consumer.py - SQLAlchemy shim · Reconciles SQLAlchemy behavior with CrateDB and MLflow expectations.
mlflow_cratedb/patch/sqlalchemy.py
External Integrations
Third-party logging and example application paths that exercise the adapter.
- PyCaret logger · Routes PyCaret MLflow logging through the compatibility layer.
mlflow_cratedb/patch/pycaret/mlflow_logger.py - PyCaret example · Runnable example showing PyCaret tracking with CrateDB-backed MLflow.
examples/tracking_pycaret.py - Tracking example · Basic tracking scenario used to demonstrate adapter usage.
examples/tracking_dummy.py
Delivery
Container and release artifacts for packaged runtime and server modes.
- Compose · Container orchestration for packaged adapter/runtime usage.
release/compose/compose.yml - OCI runtime · Runtime container image for adapter-enabled execution.
release/oci-runtime/Dockerfile - OCI server · Server-oriented container image for running the adapter with MLflow.
release/oci-server/Dockerfile
Connections 24
- CLI → Boot: starts · CLI invokes bootstrap before doing work.
- Server → Boot: starts · Server entrypoint activates bootstrap and patches.
- Boot → SQLAlchemy shim: installs · Bootstrap prepares compatibility shims.
- Boot → Tracking: activates · Bootstrap enables the MLflow tracking patch set.
- Boot → MLflow server: activates · Bootstrap enables server-side MLflow adjustments.
- Boot → DB setup: prepares · Bootstrap may initialize the CrateDB schema.
- DB setup → Create schema: applies · Setup uses create-schema DDL.
- DB setup → Drop schema: reverses · Setup uses drop-schema DDL for teardown.
- Tracking → DB utils: uses · Tracking patch relies on DB helper overrides.
- Tracking → DB types: uses · Tracking patch relies on type compatibility changes.
- Tracking → Search: uses · Tracking patch relies on patched query behavior.
- Tracking → Settings: uses · Tracking patch consults adjusted settings behavior.
- Tracking → Model: uses · Tracking patch shares model-related compatibility hooks.
- Tracking → SQLAlchemy shim: depends on · Tracking persistence sits on the SQLAlchemy shim.
- MLflow server → Tracking: coordinates · Server behavior is aligned with tracking persistence patches.
- Job runner → Huey consumer: coordinates · Job runner and consumer form the patched background execution path.
- Job runner → SQLAlchemy shim: depends on · Jobs path uses database compatibility support.
- PyCaret logger → Tracking: logs through · PyCaret integration emits MLflow tracking calls through the patch layer.
- PyCaret example → PyCaret logger: demonstrates · Example shows third-party logging integration.
- Tracking example → Tracking: demonstrates · Example exercises the tracking compatibility layer.
- Compose → Server: packages · Compose deployment targets the server entrypoint.
- Compose → Boot: packages · Compose deployment relies on bootstrap initialization.
- OCI runtime → Boot: builds · Runtime image includes bootstrap for adapter activation.
- OCI server → Server: builds · Server image wraps the server launcher.