# namhsc/mlops-crash architecture

> Architecture diagram of the GitHub repository namhsc/mlops-crash, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/namhsc/mlops-crash
- Repository: https://github.com/namhsc/mlops-crash

## Overview

1. **Repository shape**
   - This is a monorepo for an MLOps platform with four main boundaries: a **Spring Boot backend**, a **React frontend**, a **Python “core” ML/runtime layer**, and **platform/infrastructure definitions**.
   - The backend is the system of record and orchestration API; the Python core contains the actual ML pipelines and runtime services; the frontend is the operator UI; platform files assemble everything into deployable environments.

2. **Frontend boundary**
   - `frontend/` is a React 19 + TypeScript + Vite SPA.
   - It is organized by persona and workflow: general project pages, admin pages, tenant-admin pages, and user pages.
   - The UI talks only to backend API clients in `src/core/api/*`, with auth state handled in `AuthContext`, and project-scoped state in the store hooks.
   - This is the presentation layer; it does not own business data or ML execution.

3. **Backend boundary**
   - `backend/` is a Spring Boot 3.4 modular monolith using MariaDB, Spring Data JPA, and Flyway migrations.
   - The code is organized by domain modules rather than technical tiers alone: `admin`, `tenant`, `project`, `pipeline`, `deployment`, and `featurestore`.
   - Each module follows controller → service → repository patterns, with DTOs and JPA specifications for filtering/searching.
   - The backend persists core platform state, validates access, and coordinates external MLOps services.

4. **Cross-cutting backend concerns**
   - Security is split between **CAS SSO** and **JWT-based API authentication** (`CasAuthController`, `CasTicketValidationService`, `JwtAuthenticationFilter`, `SecurityConfig`).
   - Audit logging is built in via an aspect plus audit-log entity/repository/writer.
   - Sensitive fields are handled with a crypto layer (`FieldEncryptor`, `EncryptedFieldConverter`).
   - Global error handling is centralized in `GlobalExceptionHandler`, with typed exceptions and API error responses.

5. **Admin and platform-governance module**
   - `modules/admin` manages platform-wide configuration and inventory: algorithms, computing resources, packages, resource quotas, tenants, users, and system config.
   - It also contains infrastructure discovery implementations (`Local`, `Docker`, `Demo`, `Stub`) behind a factory, which suggests the backend can introspect available compute/runtime targets.
   - This module is the control plane for system administrators.

6. **Tenant-scoped data module**
   - `modules/tenant` owns data-source onboarding, dataset management, dataset versions, managed folders, tenant roles, and tenant users.
   - Supporting services include JDBC query execution, connection testing, storage access, and version analysis/comparison/publishing.
   - This is the data onboarding and governance layer feeding both feature store and training workflows.

7. **Project and collaboration module**
   - `modules/project` manages projects and project membership, acting as the collaboration boundary between users and tenant-scoped assets.
   - `ProjectMember`, `Project`, and related controllers/services imply that projects are the main unit for organizing pipelines, datasets, and deployments.
   - Access control is enforced through tenant/project membership relationships rather than a flat user model.

8. **Pipeline orchestration module**
   - `modules/pipeline` is the orchestration hub for pipeline definitions, executions, and graph-based workflow logic.
   - It contains controllers for generic pipelines plus integrations with **Airflow**, **MLflow**, and **AutoML**.
   - Core orchestration services include graph resolution, queue scheduling, tenant validation, execution helpers, and compute-provider abstractions.
   - The presence of `ComputeProvider`, resource allocation, and metrics providers indicates the backend can route work to different execution environments and monitor runtime utilization.

9. **Deployment and feature-store integration**
   - `modules/deployment` manages model-deployment records and the service layer for deployment lifecycle state.
   - `modules/featurestore` bridges the backend to **Feast**, including online feature retrieval and table/feature definition operations.
   - In the core tree, Feast configuration lives under `core/data_pipeline/feature_repo`, which indicates the feature store is managed as a separate Python-side asset but controlled from the backend.

10. **Python core: data ingestion and feature engineering**
   - `core/data_pipeline` is the ingestion/feature-materialization layer.
   - It includes Airflow DAGs for database-to-offline-store, stream-to-stores, and offline-to-online materialization.
   - The Python source is split into ingestion, cleaning, validation, processing, and utility code, suggesting a flow from raw source systems into validated offline/online feature stores.

11. **Python core: training and model lifecycle**
   - `core/training_pipeline` is the model-building layer.
   - It covers data extraction, preparation, validation, hyperparameter tuning, training, evaluation, model selection/promotion, and standalone training.
   - The template files and algorithm abstractions suggest this layer supports pluggable/custom algorithms rather than a single fixed model type.
   - Artifact storage is explicit, so trained models and metrics are persisted outside the backend database, likely in object storage such as MinIO.

12. **Python core: serving and monitoring**
   - `core/model_serving` provides batch prediction, BentoML service code, A/B testing, and data extraction helpers.
   - `core/monitoring_service` handles service monitoring, drift/performance dashboards, and retrain triggers.
   - Together these form the inference loop: model artifact promotion → serving → telemetry/monitoring → possible retraining.

13. **Infrastructure and runtime composition**
   - `platform/` contains environment-level deployment stacks for Airflow, backend, Elasticsearch/Kibana/Filebeat, Feast, Jenkins, MLflow, and Prometheus/Grafana.
   - `run.sh` is the top-level coordinator for bringing up these service stacks.
   - `core/k8s`, `core/terraform`, and the various Docker Compose files show the project supports both local containerized execution and more formal infra provisioning.

14. **End-to-end data flow**
   - Users interact through the React frontend.
   - The frontend calls the Spring Boot backend for auth, governance, project management, pipeline setup, feature-store operations, and deployment control.
   - The backend persists platform state in MariaDB and delegates execution to external systems: Airflow for orchestration, MLflow for experiment/model tracking, Feast for features, and compute/resource providers for runtime placement.
   - Python services then perform the actual data processing, training, serving, and monitoring, with artifacts and telemetry flowing through the supporting platform services.

15. **How to draw the architecture**
   - Use four major boxes: **UI**, **Backend Control Plane**, **Python ML Runtime**, and **Platform Services/Infra**.
   - Show the backend as the central API and metadata hub.
   - Show the Python core as the execution plane for pipelines, training, serving, and monitoring.
   - Show shared external services around the edge: MariaDB, Airflow, MLflow, Feast/Postgres, MinIO, Prometheus/Grafana, and ELK.

Main entry points: [`backend/src/main/java/com/vds/mlops/backend/MlopsBackendApplication.java`](https://github.com/namhsc/mlops-crash/blob/main/backend/src/main/java/com/vds/mlops/backend/MlopsBackendApplication.java) (Boot App), [`platform`](https://github.com/namhsc/mlops-crash/tree/main/platform) (Platform Stacks), [`backend/src/main/java/com/vds/mlops/backend/modules/pipeline`](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/pipeline) (Pipeline Module), [`core/mlops_common`](https://github.com/namhsc/mlops-crash/tree/main/core/mlops_common) (Shared ML Lib)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["UI"]
  node_frontend_spa["React SPA<br/>React 19/Vite UI"]
  node_frontend_api["API Clients<br/>TypeScript clients"]
  node_frontend_auth(("Auth State<br/>context/store<br/>[AuthContext.tsx]"))
  node_frontend_workflows["Persona Pages<br/>workflow pages"]
end

subgraph group_backend["Backend Control Plane"]
  node_backend_app["Boot App<br/>Spring Boot 3.4 monolith"]
  node_backend_security["Security<br/>CAS/JWT auth"]
  node_backend_common{{"Cross-Cutting<br/>audit/crypto/errors"}}
  node_backend_admin["Admin Module<br/>platform governance"]
  node_backend_tenant["Tenant Module<br/>data governance"]
  node_backend_project["Project Module<br/>collaboration boundary"]
  node_backend_pipeline["Pipeline Module<br/>orchestration hub"]
  node_backend_deployment["Deployment Module<br/>deployment records"]
  node_backend_featurestore["Feature Store<br/>Feast bridge"]
end

subgraph group_python["Python ML Runtime"]
  node_python_data["Data Pipeline<br/>Airflow/ETL"]
  node_python_train["Training Pipeline<br/>model building"]
  node_python_serving["Model Serving<br/>inference runtime"]
  node_python_monitoring["Monitoring<br/>telemetry loop"]
  node_python_common["Shared ML Lib<br/>common utilities"]
end

subgraph group_infra["Platform Services"]
  node_infra_stack["Platform Stacks<br/>compose/k8s/terraform"]
  node_infra_services[("Shared Services<br/>runtime dependencies")]
end

node_frontend_spa -->|"calls"| node_frontend_api
node_frontend_spa -->|"reads state"| node_frontend_auth
node_frontend_spa -->|"renders"| node_frontend_workflows
node_frontend_api -->|"REST"| node_backend_app
node_backend_app -->|"guards"| node_backend_security
node_backend_app -->|"uses"| node_backend_common
node_backend_app -->|"serves"| node_backend_admin
node_backend_app -->|"serves"| node_backend_tenant
node_backend_app -->|"serves"| node_backend_project
node_backend_app -->|"serves"| node_backend_pipeline
node_backend_app -->|"serves"| node_backend_deployment
node_backend_app -->|"serves"| node_backend_featurestore
node_backend_admin -->|"discovers"| node_infra_services
node_backend_tenant -->|"stores data"| node_infra_services
node_backend_pipeline -->|"coordinates"| node_infra_services
node_backend_pipeline -->|"triggers"| node_python_data
node_backend_pipeline -->|"triggers"| node_python_train
node_backend_pipeline -->|"promotes"| node_python_serving
node_backend_pipeline -->|"signals"| node_python_monitoring
node_backend_featurestore -->|"connects"| node_infra_services
node_python_data -->|"shares"| node_python_common
node_python_train -->|"shares"| node_python_common
node_python_serving -->|"shares"| node_python_common
node_python_monitoring -->|"shares"| node_python_common
node_infra_stack -->|"assembles"| node_infra_services
node_infra_stack -->|"deploys"| node_backend_app
node_infra_stack -->|"deploys"| node_python_data
node_infra_stack -->|"deploys"| node_python_train
node_infra_stack -->|"deploys"| node_python_serving
node_infra_stack -->|"deploys"| node_python_monitoring

click node_frontend_spa "https://github.com/namhsc/mlops-crash/tree/main/frontend"
click node_frontend_api "https://github.com/namhsc/mlops-crash/tree/main/frontend/src/core/api"
click node_frontend_auth "https://github.com/namhsc/mlops-crash/blob/main/frontend/src/core/contexts/AuthContext.tsx"
click node_frontend_workflows "https://github.com/namhsc/mlops-crash/tree/main/frontend/src/pages"
click node_backend_app "https://github.com/namhsc/mlops-crash/blob/main/backend/src/main/java/com/vds/mlops/backend/MlopsBackendApplication.java"
click node_backend_security "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/security"
click node_backend_common "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/common"
click node_backend_admin "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/admin"
click node_backend_tenant "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/tenant"
click node_backend_project "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/project"
click node_backend_pipeline "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/pipeline"
click node_backend_deployment "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/deployment"
click node_backend_featurestore "https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/featurestore"
click node_python_data "https://github.com/namhsc/mlops-crash/tree/main/core/data_pipeline"
click node_python_train "https://github.com/namhsc/mlops-crash/tree/main/core/training_pipeline"
click node_python_serving "https://github.com/namhsc/mlops-crash/tree/main/core/model_serving"
click node_python_monitoring "https://github.com/namhsc/mlops-crash/tree/main/core/monitoring_service"
click node_python_common "https://github.com/namhsc/mlops-crash/tree/main/core/mlops_common"
click node_infra_stack "https://github.com/namhsc/mlops-crash/tree/main/platform"
click node_infra_services "https://github.com/namhsc/mlops-crash/tree/main/platform"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_spa,node_frontend_api,node_frontend_auth,node_frontend_workflows toneBlue
class node_backend_app,node_backend_security,node_backend_common,node_backend_admin,node_backend_tenant,node_backend_project,node_backend_pipeline,node_backend_deployment,node_backend_featurestore toneAmber
class node_python_data,node_python_train,node_python_serving,node_python_monitoring,node_python_common toneMint
class node_infra_stack,node_infra_services toneRose
```

## Components

### UI

React operator console

- [React SPA](https://github.com/namhsc/mlops-crash/tree/main/frontend): React 19/Vite UI · Operator and admin presentation layer · `frontend`
- [API Clients](https://github.com/namhsc/mlops-crash/tree/main/frontend/src/core/api): TypeScript clients · Backend-facing API wrappers · `frontend/src/core/api`
- [Auth State](https://github.com/namhsc/mlops-crash/blob/main/frontend/src/core/contexts/AuthContext.tsx): context/store · Session, tenant, and project-scoped state · `frontend/src/core/contexts/AuthContext.tsx`
- [Persona Pages](https://github.com/namhsc/mlops-crash/tree/main/frontend/src/pages): workflow pages · Admin, tenant-admin, project, and user screens · `frontend/src/pages`

### Backend Control Plane

Spring Boot metadata and orchestration API

- [Boot App](https://github.com/namhsc/mlops-crash/blob/main/backend/src/main/java/com/vds/mlops/backend/MlopsBackendApplication.java): Spring Boot 3.4 monolith · System of record and API entrypoint · `backend/src/main/java/com/vds/mlops/backend/MlopsBackendApplication.java`
- [Security](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/security): CAS/JWT auth · SSO, token auth, and access enforcement · `backend/src/main/java/com/vds/mlops/backend/security`
- [Cross-Cutting](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/common): audit/crypto/errors · Audit logging, encryption, and global exceptions · `backend/src/main/java/com/vds/mlops/backend/common`
- [Admin Module](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/admin): platform governance · Algorithms, compute, packages, quotas, tenants, users, infra discovery · `backend/src/main/java/com/vds/mlops/backend/modules/admin`
- [Tenant Module](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/tenant): data governance · Data sources, datasets, versions, folders, tenant roles and users · `backend/src/main/java/com/vds/mlops/backend/modules/tenant`
- [Project Module](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/project): collaboration boundary · Projects and membership-based access control · `backend/src/main/java/com/vds/mlops/backend/modules/project`
- [Pipeline Module](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/pipeline): orchestration hub · Pipeline definitions, runs, graph resolution, queues, and provider routing · `backend/src/main/java/com/vds/mlops/backend/modules/pipeline`
- [Deployment Module](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/deployment): deployment records · Model deployment lifecycle metadata and services · `backend/src/main/java/com/vds/mlops/backend/modules/deployment`
- [Feature Store](https://github.com/namhsc/mlops-crash/tree/main/backend/src/main/java/com/vds/mlops/backend/modules/featurestore): Feast bridge · Online feature retrieval and feature/table management · `backend/src/main/java/com/vds/mlops/backend/modules/featurestore`

### Python ML Runtime

Data, training, serving, and monitoring services

- [Data Pipeline](https://github.com/namhsc/mlops-crash/tree/main/core/data_pipeline): Airflow/ETL · Ingestion, cleaning, validation, and feature materialization · `core/data_pipeline`
- [Training Pipeline](https://github.com/namhsc/mlops-crash/tree/main/core/training_pipeline): model building · Extraction, preparation, tuning, training, evaluation, and promotion · `core/training_pipeline`
- [Model Serving](https://github.com/namhsc/mlops-crash/tree/main/core/model_serving): inference runtime · Batch prediction, BentoML service, and A/B testing · `core/model_serving`
- [Monitoring](https://github.com/namhsc/mlops-crash/tree/main/core/monitoring_service): telemetry loop · Service monitoring, drift/performance dashboards, retrain triggers · `core/monitoring_service`
- [Shared ML Lib](https://github.com/namhsc/mlops-crash/tree/main/core/mlops_common): common utilities · Reusable schemas, I/O, encoders, security, and logging · `core/mlops_common`

### Platform Services

Deployable environments and shared runtime services

- [Platform Stacks](https://github.com/namhsc/mlops-crash/tree/main/platform): compose/k8s/terraform · Environment assembly for Airflow, backend, Feast, MLflow, ELK, and observability · `platform`
- [Shared Services](https://github.com/namhsc/mlops-crash/tree/main/platform): runtime dependencies · MariaDB, Airflow, MLflow, Feast/Postgres, MinIO, Prometheus/Grafana, ELK · `platform`

## Connections

- React SPA → API Clients: calls · UI uses typed backend clients
- React SPA → Auth State: reads state · Login and tenant/project context drive navigation
- React SPA → Persona Pages: renders · Persona-specific screens and flows
- API Clients → Boot App: REST · All application data flows through backend APIs
- Boot App → Security: guards · CAS SSO and JWT authentication
- Boot App → Cross-Cutting: uses · Shared audit, crypto, and error handling
- Boot App → Admin Module: serves · Platform governance endpoints
- Boot App → Tenant Module: serves · Tenant data and governance endpoints
- Boot App → Project Module: serves · Project and membership endpoints
- Boot App → Pipeline Module: serves · Pipeline orchestration endpoints
- Boot App → Deployment Module: serves · Deployment lifecycle endpoints
- Boot App → Feature Store: serves · Feature store endpoints
- Admin Module → Shared Services: discovers · Compute/runtime inventory and resource targets
- Tenant Module → Shared Services: stores data · Data source and dataset operations touch storage/DBs
- Pipeline Module → Shared Services: coordinates · Delegates orchestration, tracking, and execution placement
- Pipeline Module → Data Pipeline: triggers · Pipeline definitions and materialization jobs
- Pipeline Module → Training Pipeline: triggers · Training and tuning workflows
- Pipeline Module → Model Serving: promotes · Model deployment and batch serving integration
- Pipeline Module → Monitoring: signals · Runtime metrics and retrain feedback
- Feature Store → Shared Services: connects · Feast backend and online store dependency
- Data Pipeline → Shared ML Lib: shares · Common ingestion and validation helpers
- Training Pipeline → Shared ML Lib: shares · Common model and artifact utilities
- Model Serving → Shared ML Lib: shares · Common inference helpers and schemas
- Monitoring → Shared ML Lib: shares · Common telemetry utilities
- Platform Stacks → Shared Services: assembles · Environment definitions wire the runtime stack together
- Platform Stacks → Boot App: deploys · Backend compose stack and environment config
- Platform Stacks → Data Pipeline: deploys · Airflow/data-pipeline runtime
- Platform Stacks → Training Pipeline: deploys · Training runtime and jobs
- Platform Stacks → Model Serving: deploys · Serving runtime
- Platform Stacks → Monitoring: deploys · Monitoring runtime
