# mloda-ai/mloda architecture

> Architecture diagram of the GitHub repository mloda-ai/mloda, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mloda-ai/mloda
- Repository: https://github.com/mloda-ai/mloda

## Overview

1. **Overall shape**
   - This is a Python package that implements a **declarative data-access and feature-resolution engine**.
   - The repo is split into a **core framework** (`mloda/core`), a **plugin bundle** (`mloda_plugins`), and thin **public-facing namespaces** for users, providers, and stewards (`mloda/user`, `mloda/provider`, `mloda/steward`).

2. **Primary architectural idea**
   - Users declare **what data/features they need**.
   - mloda resolves **how to obtain and transform it** by selecting registered plugins, building dependency chains, and executing them on a compatible compute framework.
   - This makes the central boundary: **request/specification on one side, execution planning and backend-specific realization on the other**.

3. **Public API layer**
   - The user-facing API lives in `mloda/core/api` and is surfaced through `mloda.user`.
   - Key entry points include request handling, feature-config parsing/loading, and compute-framework setup.
   - The README examples show the primary usage pattern: `PluginLoader.all()` followed by `mloda.run_all(...)` with feature names, compute frameworks, and API/in-memory data.

4. **Domain model and validation**
   - `mloda/core/abstract_plugins/components` contains the shared building blocks: `Feature`, `FeatureGroup`, `ComputeFramework`, domains, options, links, filters, masks, feature sets, and data-access abstractions.
   - Validation is explicit and layered: datatype, feature-set, link, and options validators enforce invariants before execution.
   - This is where the repository defines the **contract** that plugins must satisfy.

5. **Planning and resolution pipeline**
   - The `mloda/core/prepare` package is the orchestration/planning subsystem.
   - It builds the execution graph, identifies feature groups, resolves links, resolves compute frameworks, and converts a declarative request into an execution plan.
   - The internal graph/step model in `mloda/core/prepare/graph` and `mloda/core/core/step` is the main intermediate representation for diagrams: request → graph resolution → ordered steps.

6. **Runtime execution**
   - `mloda/core/runtime` executes the plan using workers, a data lifecycle manager, and a compute-framework executor.
   - Concurrency is abstracted through thread and multiprocessing workers.
   - There is also a **Flight server boundary** (`runtime/flight`) for Arrow Flight-based serving/transport, which is the main network/service-style runtime component in the repo.

7. **Compute-framework abstraction**
   - Compute frameworks are first-class plugins: pandas, Polars, PyArrow, Python dict, DuckDB, SQLite, Spark, and Iceberg all appear as implementations under `mloda_plugins/compute_framework/base_implementations`.
   - Each framework typically provides framework-specific filter, mask, merge, transformer, and relation objects.
   - The important architectural boundary is that **the core engine reasons over abstract operations**, while each framework supplies the concrete data-structure behavior.

8. **Feature-group plugins**
   - Most domain behavior lives in `mloda_plugins/feature_group`.
   - These plugins implement source readers, transformations, enrichments, aggregations, time windows, text cleaning, missing-value handling, sklearn-style transformations, forecasting, graph metrics, and other experimental feature groups.
   - These are the units that define “what can be requested” and how dependencies chain together.

9. **Input data and ingestion**
   - Input-related plugins under `mloda_plugins/feature_group/input_data` and `mloda/core/abstract_plugins/components/input_data` handle files, databases, documents, context files, and API-style data.
   - The README’s `api_data` example and the `docs/docs/examples` folder show that in-memory data is also a first-class source.
   - In architecture terms, this is the **source boundary**: external files, DBs, APIs, and in-memory payloads enter through input-data plugins, not through ad hoc code paths.

10. **Plugin discovery and policy**
   - `mloda/core/abstract_plugins/plugin_loader` and `plugin_registry` provide discovery, registration, and policy enforcement.
   - Tests show support for entry points, strict mode, loader registration, and registry injection, so plugin loading is not informal—it is governed by a registry/policy layer.
   - This is a key control plane for the system: **available plugins determine the resolution outcomes**.

11. **Extensibility and observability hooks**
   - `mloda/core/abstract_plugins/function_extender` and `mloda_plugins/function_extender/base_implementations/otel` provide wrapper-style hooks around execution.
   - Extenders are used for logging, validation, tracing, and monitoring without embedding those concerns in feature groups or compute frameworks.
   - In an architecture diagram, these are cross-cutting wrappers around feature execution.

12. **Dataflow between subsystems**
   - Typical flow: user request or config → feature parsing → plugin discovery → graph/link resolution → compute-framework resolution → execution plan → runtime workers → backend-specific operations → result aggregation.
   - Cross-framework data conversion is handled via Arrow-oriented transformers in many plugins, which is how the system moves data between Pandas/Polars/Spark/SQL-like backends.

13. **Repository-level packaging and delivery**
   - The repo is a standard Python distribution (`pyproject.toml`, `MANIFEST.in`, `uv.lock`) with typed package markers (`py.typed`).
   - CI/CD is handled by GitHub Actions, release automation, and security scanning.
   - Documentation is a major artifact (`docs/` with MkDocs), and `tests/` mirrors almost every subsystem, indicating the architecture is heavily contract-tested.

14. **What to draw as the main boundaries**
   - For an architecture diagram, the most important boxes are:
     - **User/API boundary**: feature requests and config parsing.
     - **Planning boundary**: graph construction, link and framework resolution.
     - **Runtime boundary**: execution orchestration, workers, Flight server.
     - **Plugin boundary**: feature groups, compute frameworks, extenders.
     - **Source boundary**: files, databases, APIs, in-memory data.
   - The core design is a **plugin-resolved execution engine** rather than a single fixed pipeline or monolithic backend.

Main entry points: [`mloda/core/api/request.py`](https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/request.py) (request API), [`mloda/core/abstract_plugins/plugin_registry/plugin_registry.py`](https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/plugin_registry/plugin_registry.py) (registry policy), [`mloda/core/runtime/run.py`](https://github.com/mloda-ai/mloda/blob/main/mloda/core/runtime/run.py) (runtime), [`mloda/core/prepare/graph/build_graph.py`](https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/graph/build_graph.py) (plan graph)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public["Public API"]
  node_mloda_user["user<br/>public namespace<br/>[__init__.py]"]
  node_mloda_provider["provider<br/>public namespace<br/>[__init__.py]"]
  node_mloda_steward["steward<br/>public namespace<br/>[__init__.py]"]
end

subgraph group_core["Core engine"]
  node_core_api_request["request API<br/>entrypoint<br/>[request.py]"]
  node_core_api_feature_config["feature config<br/>config API<br/>[parser.py]"]
  node_core_api_prepare["setup CF<br/>prepare API"]
  node_core_components["domain model<br/>contracts<br/>[feature.py]"]
  node_core_plugin_loader["plugin loader<br/>discovery<br/>[plugin_loader.py]"]
  node_core_plugin_registry["registry policy<br/>[plugin_registry.py]"]
  node_core_prepare_graph["plan graph<br/>planning IR<br/>[build_graph.py]"]
  node_core_prepare_resolve["resolve plan<br/>planner<br/>[resolve_graph.py]"]
  node_core_execution_plan["exec plan<br/>[execution_plan.py]"]
  node_core_engine["engine<br/>orchestrator<br/>[engine.py]"]
  node_core_runtime["runtime<br/>executor<br/>[run.py]"]
  node_core_flight["flight server<br/>transport<br/>[flight_server.py]"]
end

subgraph group_plugins["Plugins"]
  node_plugins_feature_groups["feature groups<br/>plugin bundle<br/>[__init__.py]"]
  node_plugins_compute_frameworks["compute frameworks<br/>plugin bundle<br/>[__init__.py]"]
  node_plugins_extender["extenders<br/>cross-cutting plugin<br/>[otel_extender.py]"]
end

node_mloda_user -->|"uses"| node_core_api_request
node_core_api_request -->|"parses"| node_core_api_feature_config
node_core_api_request -->|"sets up"| node_core_api_prepare
node_core_api_request -->|"validates against"| node_core_components
node_core_api_request -->|"loads"| node_core_plugin_loader
node_core_plugin_loader -->|"consults"| node_core_plugin_registry
node_core_plugin_registry -->|"exposes"| node_plugins_feature_groups
node_core_plugin_registry -->|"exposes"| node_plugins_compute_frameworks
node_core_plugin_registry -->|"exposes"| node_plugins_extender
node_core_components -->|"feeds"| node_core_prepare_graph
node_plugins_feature_groups -->|"contributes"| node_core_prepare_graph
node_core_prepare_graph -->|"builds"| node_core_prepare_resolve
node_core_prepare_resolve -->|"materializes"| node_core_execution_plan
node_core_execution_plan -->|"runs in"| node_core_engine
node_core_engine -->|"dispatches"| node_core_runtime
node_core_runtime -->|"executes on"| node_plugins_compute_frameworks
node_core_runtime -->|"wraps with"| node_plugins_extender
node_core_runtime -->|"serves via"| node_core_flight
node_core_prepare_resolve -->|"selects"| node_plugins_compute_frameworks
node_core_api_feature_config -->|"supplies"| node_core_prepare_graph
node_mloda_provider -->|"implements"| node_core_components
node_mloda_steward -->|"governs"| node_core_plugin_registry

click node_mloda_user "https://github.com/mloda-ai/mloda/blob/main/mloda/user/__init__.py"
click node_mloda_provider "https://github.com/mloda-ai/mloda/blob/main/mloda/provider/__init__.py"
click node_mloda_steward "https://github.com/mloda-ai/mloda/blob/main/mloda/steward/__init__.py"
click node_core_api_request "https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/request.py"
click node_core_api_feature_config "https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/feature_config/parser.py"
click node_core_api_prepare "https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/prepare/setup_compute_framework.py"
click node_core_components "https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/components/feature.py"
click node_core_plugin_loader "https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/plugin_loader/plugin_loader.py"
click node_core_plugin_registry "https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/plugin_registry/plugin_registry.py"
click node_core_prepare_graph "https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/graph/build_graph.py"
click node_core_prepare_resolve "https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/resolve_graph.py"
click node_core_execution_plan "https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/execution_plan.py"
click node_core_engine "https://github.com/mloda-ai/mloda/blob/main/mloda/core/core/engine.py"
click node_core_runtime "https://github.com/mloda-ai/mloda/blob/main/mloda/core/runtime/run.py"
click node_core_flight "https://github.com/mloda-ai/mloda/blob/main/mloda/core/runtime/flight/flight_server.py"
click node_plugins_feature_groups "https://github.com/mloda-ai/mloda/blob/main/mloda_plugins/feature_group/__init__.py"
click node_plugins_compute_frameworks "https://github.com/mloda-ai/mloda/blob/main/mloda_plugins/compute_framework/base_implementations/__init__.py"
click node_plugins_extender "https://github.com/mloda-ai/mloda/blob/main/mloda_plugins/function_extender/base_implementations/otel/otel_extender.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mloda_user,node_mloda_provider,node_mloda_steward toneBlue
class node_core_api_request,node_core_api_feature_config,node_core_api_prepare,node_core_components,node_core_plugin_loader,node_core_plugin_registry,node_core_prepare_graph,node_core_prepare_resolve,node_core_execution_plan,node_core_engine,node_core_runtime,node_core_flight toneAmber
class node_plugins_feature_groups,node_plugins_compute_frameworks,node_plugins_extender toneMint
```

## Components

### Public API

User-facing namespaces and entrypoints

- [user](https://github.com/mloda-ai/mloda/blob/main/mloda/user/__init__.py): public namespace · User-facing package surface · `mloda/user/__init__.py`
- [provider](https://github.com/mloda-ai/mloda/blob/main/mloda/provider/__init__.py): public namespace · Provider-facing package surface · `mloda/provider/__init__.py`
- [steward](https://github.com/mloda-ai/mloda/blob/main/mloda/steward/__init__.py): public namespace · Steward-facing package surface · `mloda/steward/__init__.py`

### Core engine

Declarative request handling, planning, and runtime execution

- [request API](https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/request.py): entrypoint · Run and request entrypoints · `mloda/core/api/request.py`
- [feature config](https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/feature_config/parser.py): config API · Feature config parsing and loading · `mloda/core/api/feature_config/parser.py`
- [setup CF](https://github.com/mloda-ai/mloda/blob/main/mloda/core/api/prepare/setup_compute_framework.py): prepare API · Compute-framework setup helpers · `mloda/core/api/prepare/setup_compute_framework.py`
- [domain model](https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/components/feature.py): contracts · Features, links, options, filters, data access, and validators · `mloda/core/abstract_plugins/components/feature.py`
- [plugin loader](https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/plugin_loader/plugin_loader.py): discovery · Plugin discovery and registration control plane · `mloda/core/abstract_plugins/plugin_loader/plugin_loader.py`
- [registry policy](https://github.com/mloda-ai/mloda/blob/main/mloda/core/abstract_plugins/plugin_registry/plugin_registry.py): policy · Registry state and policy enforcement · `mloda/core/abstract_plugins/plugin_registry/plugin_registry.py`
- [plan graph](https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/graph/build_graph.py): planning IR · Graph construction and resolution intermediates · `mloda/core/prepare/graph/build_graph.py`
- [resolve plan](https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/resolve_graph.py): planner · Feature-group, link, and compute-framework resolution · `mloda/core/prepare/resolve_graph.py`
- [exec plan](https://github.com/mloda-ai/mloda/blob/main/mloda/core/prepare/execution_plan.py): plan · Ordered execution plan and steps · `mloda/core/prepare/execution_plan.py`
- [engine](https://github.com/mloda-ai/mloda/blob/main/mloda/core/core/engine.py): orchestrator · Core execution coordination · `mloda/core/core/engine.py`
- [runtime](https://github.com/mloda-ai/mloda/blob/main/mloda/core/runtime/run.py): executor · Workers, lifecycle, and backend execution · `mloda/core/runtime/run.py`
- [flight server](https://github.com/mloda-ai/mloda/blob/main/mloda/core/runtime/flight/flight_server.py): transport · Arrow Flight serving boundary · `mloda/core/runtime/flight/flight_server.py`

### Plugins

Feature groups, compute frameworks, and extenders

- [feature groups](https://github.com/mloda-ai/mloda/blob/main/mloda_plugins/feature_group/__init__.py): plugin bundle · Source, transform, enrichment, and experimental feature plugins · `mloda_plugins/feature_group/__init__.py`
- [compute frameworks](https://github.com/mloda-ai/mloda/blob/main/mloda_plugins/compute_framework/base_implementations/__init__.py): plugin bundle · Backend-specific compute implementations · `mloda_plugins/compute_framework/base_implementations/__init__.py`
- [extenders](https://github.com/mloda-ai/mloda/blob/main/mloda_plugins/function_extender/base_implementations/otel/otel_extender.py): cross-cutting plugin · Tracing, logging, and monitoring wrappers · `mloda_plugins/function_extender/base_implementations/otel/otel_extender.py`

## Connections

- user → request API: uses · User namespace surfaces the request entrypoint
- request API → feature config: parses · Requests may include declarative feature config
- request API → setup CF: sets up · API prepares compute-framework execution
- request API → domain model: validates against · Requests are checked against domain contracts
- request API → plugin loader: loads · Entry flow discovers available plugins
- plugin loader → registry policy: consults · Discovery is governed by registry policy
- registry policy → feature groups: exposes · Registered feature-group plugins become resolvable
- registry policy → compute frameworks: exposes · Registered compute frameworks become selectable
- registry policy → extenders: exposes · Registered extenders wrap execution
- domain model → plan graph: feeds · Domain objects define the planning inputs
- feature groups → plan graph: contributes · Feature groups become nodes in the plan graph
- plan graph → resolve plan: builds · Graph is resolved into execution dependencies
- resolve plan → exec plan: materializes · Resolution becomes an ordered execution plan
- exec plan → engine: runs in · Engine orchestrates plan execution
- engine → runtime: dispatches · Runtime handles worker execution
- runtime → compute frameworks: executes on · Backend operations are delegated to framework plugins
- runtime → extenders: wraps with · Cross-cutting extenders observe or augment execution
- runtime → flight server: serves via · Runtime can expose an Arrow Flight boundary
- resolve plan → compute frameworks: selects · Planner chooses compatible frameworks
- feature config → plan graph: supplies · Parsed config feeds graph construction
- provider → domain model: implements · Provider surface aligns with shared contracts
- steward → registry policy: governs · Steward surface maps to policy and control