mloda-ai/mloda

1. Overall shape

- This is a Python package that implements a declarative data-access and feature-resolution engine.

- The repo is split into a core framework (mloda/core), a plugin bundle (mloda_plugins), and thin public-facing namespaces for users, providers, and stewards (mloda/user, mloda/provider, mloda/steward).

2. Primary architectural idea

- Users declare what data/features they need.

- mloda resolves how to obtain and transform it by selecting registered plugins, building dependency chains, and executing them on a compatible compute framework.

- This makes the central boundary: request/specification on one side, execution planning and backend-specific realization on the other.

3. Public API layer

- The user-facing API lives in mloda/core/api and is surfaced through mloda.user.

- Key entry points include request handling, feature-config parsing/loading, and compute-framework setup.

- The README examples show the primary usage pattern: PluginLoader.all() followed by mloda.run_all(...) with feature names, compute frameworks, and API/in-memory data.

4. Domain model and validation

- mloda/core/abstract_plugins/components contains the shared building blocks: Feature, FeatureGroup, ComputeFramework, domains, options, links, filters, masks, feature sets, and data-access abstractions.

- Validation is explicit and layered: datatype, feature-set, link, and options validators enforce invariants before execution.

- This is where the repository defines the contract that plugins must satisfy.

5. Planning and resolution pipeline

- The mloda/core/prepare package is the orchestration/planning subsystem.

- It builds the execution graph, identifies feature groups, resolves links, resolves compute frameworks, and converts a declarative request into an execution plan.

- The internal graph/step model in mloda/core/prepare/graph and mloda/core/core/step is the main intermediate representation for diagrams: request → graph resolution → ordered steps.

6. Runtime execution

- mloda/core/runtime executes the plan using workers, a data lifecycle manager, and a compute-framework executor.

- Concurrency is abstracted through thread and multiprocessing workers.

- There is also a Flight server boundary (runtime/flight) for Arrow Flight-based serving/transport, which is the main network/service-style runtime component in the repo.

7. Compute-framework abstraction

- Compute frameworks are first-class plugins: pandas, Polars, PyArrow, Python dict, DuckDB, SQLite, Spark, and Iceberg all appear as implementations under mloda_plugins/compute_framework/base_implementations.

- Each framework typically provides framework-specific filter, mask, merge, transformer, and relation objects.

- The important architectural boundary is that the core engine reasons over abstract operations, while each framework supplies the concrete data-structure behavior.

8. Feature-group plugins

- Most domain behavior lives in mloda_plugins/feature_group.

- These plugins implement source readers, transformations, enrichments, aggregations, time windows, text cleaning, missing-value handling, sklearn-style transformations, forecasting, graph metrics, and other experimental feature groups.

- These are the units that define “what can be requested” and how dependencies chain together.

9. Input data and ingestion

- Input-related plugins under mloda_plugins/feature_group/input_data and mloda/core/abstract_plugins/components/input_data handle files, databases, documents, context files, and API-style data.

- The README’s api_data example and the docs/docs/examples folder show that in-memory data is also a first-class source.

- In architecture terms, this is the source boundary: external files, DBs, APIs, and in-memory payloads enter through input-data plugins, not through ad hoc code paths.

10. Plugin discovery and policy

- mloda/core/abstract_plugins/plugin_loader and plugin_registry provide discovery, registration, and policy enforcement.

- Tests show support for entry points, strict mode, loader registration, and registry injection, so plugin loading is not informal—it is governed by a registry/policy layer.

- This is a key control plane for the system: available plugins determine the resolution outcomes.

11. Extensibility and observability hooks

- mloda/core/abstract_plugins/function_extender and mloda_plugins/function_extender/base_implementations/otel provide wrapper-style hooks around execution.

- Extenders are used for logging, validation, tracing, and monitoring without embedding those concerns in feature groups or compute frameworks.

- In an architecture diagram, these are cross-cutting wrappers around feature execution.

12. Dataflow between subsystems

- Typical flow: user request or config → feature parsing → plugin discovery → graph/link resolution → compute-framework resolution → execution plan → runtime workers → backend-specific operations → result aggregation.

- Cross-framework data conversion is handled via Arrow-oriented transformers in many plugins, which is how the system moves data between Pandas/Polars/Spark/SQL-like backends.

13. Repository-level packaging and delivery

- The repo is a standard Python distribution (pyproject.toml, MANIFEST.in, uv.lock) with typed package markers (py.typed).

- CI/CD is handled by GitHub Actions, release automation, and security scanning.

- Documentation is a major artifact (docs/ with MkDocs), and tests/ mirrors almost every subsystem, indicating the architecture is heavily contract-tested.

14. What to draw as the main boundaries

- For an architecture diagram, the most important boxes are:

- User/API boundary: feature requests and config parsing.

- Planning boundary: graph construction, link and framework resolution.

- Runtime boundary: execution orchestration, workers, Flight server.

- Plugin boundary: feature groups, compute frameworks, extenders.

- Source boundary: files, databases, APIs, in-memory data.

- The core design is a plugin-resolved execution engine rather than a single fixed pipeline or monolithic backend.