# dfds-data/mlflow-infrastructure architecture

> Architecture diagram of the GitHub repository dfds-data/mlflow-infrastructure, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dfds-data/mlflow-infrastructure
- Repository: https://github.com/dfds-data/mlflow-infrastructure

## Overview

##### Project type and purpose
- This is an infrastructure‐as‐code repository for centrally managed MLflow deployments.
- “/infrastructure” defines Terraform for a shared PostgreSQL backend in a central AWS account.
- “/mlflow-image” builds and publishes versioned Docker images of MLflow to ECR.
- “/mlflow-terraform” provides Terraform modules for client teams to provision S3 buckets and IAM roles in their AWS accounts so their Kubernetes‐hosted MLflow servers can store artifacts.

##### File structure
- Top‐level CI/CD configurations:
- .github/workflows/check-mlflow-release.yml (checks MLflow versions)
- azure-pipelines.yml (build & push MLflow Docker images, apply central Terraform)
- infrastructure/ (central AWS account Terraform for PostgreSQL RDS)
- mlflow-image/ (Dockerfile, docker-compose, pipeline scripts for image build & push to ECR)
- mlflow-terraform/ (client‐side Terraform to create S3 bucket & IAM role for MLflow artifact store)

##### README for additional insights
- Central DB to avoid per-project servers
- Automated version detection via GitHub Actions → PR → on-merge triggers Azure Pipelines
- Clients directly reference source code of Terraform modules for their releases

##### Components
a. Version Management & CI/CD
- GitHub repo (with Infra, Image, Terraform modules)
- GitHub Actions (daily MLflow version check)
- Azure Pipelines (on‐merge build & deploy)
b. Container Registry
- AWS ECR (shared repository for MLflow images)
c. Central AWS Account (managed by “/infrastructure”)
- PostgreSQL RDS instance (Terraform)
d. Client AWS Accounts (via “/mlflow-terraform”)
- S3 Bucket (artifact storage)
- IAM Role (assumed by K8s)
e. Kubernetes Clusters (outside this repo, per client)
- Deploy MLflow server pods (pull image from ECR, connect to central RDS, write artifacts to S3 via IAM)

##### Relationships and data flows
- Build Docker image → Push to ECR
- Apply “/infrastructure” Terraform → Create/Update RDS
- Client team runs “/mlflow-terraform” → Terraform applies in their AWS → S3 + IAM
- K8s cluster pulls MLflow image from ECR, connects to RDS (TCP), writes artifacts to S3 (via IAM)

##### Architectural patterns and design principles
- Centralized shared‐backend for MLflow state (avoid duplicate RDS)
- Module‐based Terraform for reusability (central vs. client modules)
- GitOps/CI‐driven automation for releases and infra apply
- Containerization of MLflow for uniform deployment

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Version Management & CI/CD"
        direction TB
        Repo["GitHub Repo"]:::ci
        Actions["GitHub Actions"]:::ci
        Pipelines["Azure Pipelines"]:::ci
        MLflowImage["mlflow-image\n(Docker build & push)"]:::ci
    end

    subgraph "Central AWS Account" 
        direction TB
        ECR["AWS ECR\n(Container Registry)"]:::central
        RDS["PostgreSQL RDS\n(Database)"]:::central
    end

    subgraph "Client AWS Account"
        direction TB
        MLflowTF["mlflow-terraform\n(Terraform module)"]:::client
        S3["S3 Bucket\n(Artifact Storage)"]:::client
        IAM["IAM Role\n(K8s Access)"]:::client
    end

    subgraph "Kubernetes Clusters"
        direction TB
        K8s["K8s Cluster\n(MLflow Server)"]:::k8s
    end

    Actions -->|"checks MLflow version"| Repo
    Repo -->|"on merge"| Pipelines
    Pipelines -->|"docker build"| MLflowImage
    MLflowImage -->|"docker push"| ECR
    Pipelines -->|"terraform apply"| RDS

    MLflowTF -->|"terraform apply"| S3
    MLflowTF -->|"terraform apply"| IAM

    K8s -->|"docker pull"| ECR
    K8s -->|"TCP connect"| RDS
    K8s -->|"S3 PUT via IAM"| S3

    classDef ci fill:#c2f0c2,stroke:#2d862d,font-weight:bold
    classDef central fill:#c2d9f8,stroke:#1f4e79,font-weight:bold
    classDef client fill:#fce5cd,stroke:#b45f06,font-weight:bold
    classDef k8s fill:#e8daef,stroke:#5b2c6f,font-weight:bold

    click Actions "https://github.com/dfds-data/mlflow-infrastructure/blob/master/.github/workflows/check-mlflow-release.yml"
    click Pipelines "https://github.com/dfds-data/mlflow-infrastructure/blob/master/azure-pipelines.yml"
    click MLflowImage "https://github.com/dfds-data/mlflow-infrastructure/tree/master/mlflow-image/"
    click Repo "https://github.com/dfds-data/mlflow-infrastructure/blob/master/."
    click MLflowTF "https://github.com/dfds-data/mlflow-infrastructure/tree/master/mlflow-terraform/"
    click ECR "https://github.com/dfds-data/mlflow-infrastructure/tree/master/mlflow-image/"
    click RDS "https://github.com/dfds-data/mlflow-infrastructure/tree/master/infrastructure/"
    click S3 "https://github.com/dfds-data/mlflow-infrastructure/tree/master/mlflow-terraform/"
    click IAM "https://github.com/dfds-data/mlflow-infrastructure/tree/master/mlflow-terraform/"
```
