How namhsc/mlops-crash works
1. Repository shape
- This is a monorepo for an MLOps platform with four main boundaries: a Spring Boot backend, a React frontend, a Python “core” ML/runtime layer, and platform/infrastructure definitions.
- The backend is the system of record and orchestration API; the Python core contains the actual ML pipelines and runtime services; the frontend is the operator UI; platform files assemble everything into deployable environments.
2. Frontend boundary
- frontend/ is a React 19 + TypeScript + Vite SPA.
- It is organized by persona and workflow: general project pages, admin pages, tenant-admin pages, and user pages.
- The UI talks only to backend API clients in src/core/api/*, with auth state handled in AuthContext, and project-scoped state in the store hooks.
- This is the presentation layer; it does not own business data or ML execution.
3. Backend boundary
- backend/ is a Spring Boot 3.4 modular monolith using MariaDB, Spring Data JPA, and Flyway migrations.
- The code is organized by domain modules rather than technical tiers alone: admin, tenant, project, pipeline, deployment, and featurestore.
- Each module follows controller → service → repository patterns, with DTOs and JPA specifications for filtering/searching.
- The backend persists core platform state, validates access, and coordinates external MLOps services.
4. Cross-cutting backend concerns
- Security is split between CAS SSO and JWT-based API authentication (CasAuthController, CasTicketValidationService, JwtAuthenticationFilter, SecurityConfig).
- Audit logging is built in via an aspect plus audit-log entity/repository/writer.
- Sensitive fields are handled with a crypto layer (FieldEncryptor, EncryptedFieldConverter).
- Global error handling is centralized in GlobalExceptionHandler, with typed exceptions and API error responses.
5. Admin and platform-governance module
- modules/admin manages platform-wide configuration and inventory: algorithms, computing resources, packages, resource quotas, tenants, users, and system config.
- It also contains infrastructure discovery implementations (Local, Docker, Demo, Stub) behind a factory, which suggests the backend can introspect available compute/runtime targets.
- This module is the control plane for system administrators.
6. Tenant-scoped data module
- modules/tenant owns data-source onboarding, dataset management, dataset versions, managed folders, tenant roles, and tenant users.
- Supporting services include JDBC query execution, connection testing, storage access, and version analysis/comparison/publishing.
- This is the data onboarding and governance layer feeding both feature store and training workflows.
7. Project and collaboration module
- modules/project manages projects and project membership, acting as the collaboration boundary between users and tenant-scoped assets.
- ProjectMember, Project, and related controllers/services imply that projects are the main unit for organizing pipelines, datasets, and deployments.
- Access control is enforced through tenant/project membership relationships rather than a flat user model.
8. Pipeline orchestration module
- modules/pipeline is the orchestration hub for pipeline definitions, executions, and graph-based workflow logic.
- It contains controllers for generic pipelines plus integrations with Airflow, MLflow, and AutoML.
- Core orchestration services include graph resolution, queue scheduling, tenant validation, execution helpers, and compute-provider abstractions.
- The presence of ComputeProvider, resource allocation, and metrics providers indicates the backend can route work to different execution environments and monitor runtime utilization.
9. Deployment and feature-store integration
- modules/deployment manages model-deployment records and the service layer for deployment lifecycle state.
- modules/featurestore bridges the backend to Feast, including online feature retrieval and table/feature definition operations.
- In the core tree, Feast configuration lives under core/data_pipeline/feature_repo, which indicates the feature store is managed as a separate Python-side asset but controlled from the backend.
10. Python core: data ingestion and feature engineering
- core/data_pipeline is the ingestion/feature-materialization layer.
- It includes Airflow DAGs for database-to-offline-store, stream-to-stores, and offline-to-online materialization.
- The Python source is split into ingestion, cleaning, validation, processing, and utility code, suggesting a flow from raw source systems into validated offline/online feature stores.
11. Python core: training and model lifecycle
- core/training_pipeline is the model-building layer.
- It covers data extraction, preparation, validation, hyperparameter tuning, training, evaluation, model selection/promotion, and standalone training.
- The template files and algorithm abstractions suggest this layer supports pluggable/custom algorithms rather than a single fixed model type.
- Artifact storage is explicit, so trained models and metrics are persisted outside the backend database, likely in object storage such as MinIO.
12. Python core: serving and monitoring
- core/model_serving provides batch prediction, BentoML service code, A/B testing, and data extraction helpers.
- core/monitoring_service handles service monitoring, drift/performance dashboards, and retrain triggers.
- Together these form the inference loop: model artifact promotion → serving → telemetry/monitoring → possible retraining.
13. Infrastructure and runtime composition
- platform/ contains environment-level deployment stacks for Airflow, backend, Elasticsearch/Kibana/Filebeat, Feast, Jenkins, MLflow, and Prometheus/Grafana.
- run.sh is the top-level coordinator for bringing up these service stacks.
- core/k8s, core/terraform, and the various Docker Compose files show the project supports both local containerized execution and more formal infra provisioning.
14. End-to-end data flow
- Users interact through the React frontend.
- The frontend calls the Spring Boot backend for auth, governance, project management, pipeline setup, feature-store operations, and deployment control.
- The backend persists platform state in MariaDB and delegates execution to external systems: Airflow for orchestration, MLflow for experiment/model tracking, Feast for features, and compute/resource providers for runtime placement.
- Python services then perform the actual data processing, training, serving, and monitoring, with artifacts and telemetry flowing through the supporting platform services.
15. How to draw the architecture
- Use four major boxes: UI, Backend Control Plane, Python ML Runtime, and Platform Services/Infra.
- Show the backend as the central API and metadata hub.
- Show the Python core as the execution plane for pipelines, training, serving, and monitoring.
- Show shared external services around the edge: MariaDB, Airflow, MLflow, Feast/Postgres, MinIO, Prometheus/Grafana, and ELK.
Main entry points: MlopsBackendApplication.java Boot App · platform Platform Stacks · pipeline Pipeline Module · mlops_common Shared ML Lib
Components 20
UI
React operator console
- React SPA · Operator and admin presentation layer
frontend - API Clients · Backend-facing API wrappers
frontend/src/core/api - Auth State · Session, tenant, and project-scoped state
frontend/src/core/contexts/AuthContext.tsx - Persona Pages · Admin, tenant-admin, project, and user screens
frontend/src/pages
Backend Control Plane
Spring Boot metadata and orchestration API
- Boot App · System of record and API entrypoint
backend/src/main/java/com/vds/mlops/backend/MlopsBackendApplication.java - Security · SSO, token auth, and access enforcement
backend/src/main/java/com/vds/mlops/backend/security - Cross-Cutting · Audit logging, encryption, and global exceptions
backend/src/main/java/com/vds/mlops/backend/common - Admin Module · Algorithms, compute, packages, quotas, tenants, users, infra discovery
backend/src/main/java/com/vds/mlops/backend/modules/admin - Tenant Module · Data sources, datasets, versions, folders, tenant roles and users
backend/src/main/java/com/vds/mlops/backend/modules/tenant - Project Module · Projects and membership-based access control
backend/src/main/java/com/vds/mlops/backend/modules/project - Pipeline Module · Pipeline definitions, runs, graph resolution, queues, and provider routing
backend/src/main/java/com/vds/mlops/backend/modules/pipeline - Deployment Module · Model deployment lifecycle metadata and services
backend/src/main/java/com/vds/mlops/backend/modules/deployment - Feature Store · Online feature retrieval and feature/table management
backend/src/main/java/com/vds/mlops/backend/modules/featurestore
Python ML Runtime
Data, training, serving, and monitoring services
- Data Pipeline · Ingestion, cleaning, validation, and feature materialization
core/data_pipeline - Training Pipeline · Extraction, preparation, tuning, training, evaluation, and promotion
core/training_pipeline - Model Serving · Batch prediction, BentoML service, and A/B testing
core/model_serving - Monitoring · Service monitoring, drift/performance dashboards, retrain triggers
core/monitoring_service - Shared ML Lib · Reusable schemas, I/O, encoders, security, and logging
core/mlops_common
Platform Services
Deployable environments and shared runtime services
- Platform Stacks · Environment assembly for Airflow, backend, Feast, MLflow, ELK, and observability
platform - Shared Services · MariaDB, Airflow, MLflow, Feast/Postgres, MinIO, Prometheus/Grafana, ELK
platform
Connections 30
- React SPA → API Clients: calls · UI uses typed backend clients
- React SPA → Auth State: reads state · Login and tenant/project context drive navigation
- React SPA → Persona Pages: renders · Persona-specific screens and flows
- API Clients → Boot App: REST · All application data flows through backend APIs
- Boot App → Security: guards · CAS SSO and JWT authentication
- Boot App → Cross-Cutting: uses · Shared audit, crypto, and error handling
- Boot App → Admin Module: serves · Platform governance endpoints
- Boot App → Tenant Module: serves · Tenant data and governance endpoints
- Boot App → Project Module: serves · Project and membership endpoints
- Boot App → Pipeline Module: serves · Pipeline orchestration endpoints
- Boot App → Deployment Module: serves · Deployment lifecycle endpoints
- Boot App → Feature Store: serves · Feature store endpoints
- Admin Module → Shared Services: discovers · Compute/runtime inventory and resource targets
- Tenant Module → Shared Services: stores data · Data source and dataset operations touch storage/DBs
- Pipeline Module → Shared Services: coordinates · Delegates orchestration, tracking, and execution placement
- Pipeline Module → Data Pipeline: triggers · Pipeline definitions and materialization jobs
- Pipeline Module → Training Pipeline: triggers · Training and tuning workflows
- Pipeline Module → Model Serving: promotes · Model deployment and batch serving integration
- Pipeline Module → Monitoring: signals · Runtime metrics and retrain feedback
- Feature Store → Shared Services: connects · Feast backend and online store dependency
- Data Pipeline → Shared ML Lib: shares · Common ingestion and validation helpers
- Training Pipeline → Shared ML Lib: shares · Common model and artifact utilities
- Model Serving → Shared ML Lib: shares · Common inference helpers and schemas
- Monitoring → Shared ML Lib: shares · Common telemetry utilities
- Platform Stacks → Shared Services: assembles · Environment definitions wire the runtime stack together
- Platform Stacks → Boot App: deploys · Backend compose stack and environment config
- Platform Stacks → Data Pipeline: deploys · Airflow/data-pipeline runtime
- Platform Stacks → Training Pipeline: deploys · Training runtime and jobs
- Platform Stacks → Model Serving: deploys · Serving runtime
- Platform Stacks → Monitoring: deploys · Monitoring runtime