# vincentschaik/production architecture

> Architecture diagram of the GitHub repository vincentschaik/production, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vincentschaik/production
- Repository: https://github.com/vincentschaik/production

## Overview

##### Project type and purpose
- This is an educational “ML Systems Design” module repository rather than a customer-facing application.
- It bundles slide decks and Jupyter labs (folders 01_materials, 02_activities, 03_instructional_team, 04_this_cohort) plus a demo source tree (05_src) that spins up a mini production-like experiment-tracking backend via Docker.

##### File structure
- Top level:
- .github: PR/issue templates and workflows
- 01_materials: Live-lab notebooks & PDF slides
- 02_activities: Graded assignments & rubrics
- 03_instructional_team: Markdown slides + slide-generator script
- 04_this_cohort: Cohort-specific extras
- 05_src: Demo code + infra for ML pipelines
- Inside 05_src:
- credit/: data ingestion and three logistic-regression experiments
- stock_prices/: data_manager for another demo
- experiment_tracking/: docker-compose setup for MLflow, MinIO, PostgreSQL + test script
- utils/: shared logging utility
- Config: environment.yaml, .env files, docker-compose.yml, Dockerfiles, SQL init scripts.

##### README for additional insights
- The module covers data engineering, feature engineering, hyperparameter tuning, deployment, explainability, experiment tracking/monitoring.
- Students will run Docker to mimic production (Postgres + S3-style storage via MinIO) and use MLflow for experiment tracking.

a. Main components
- Learner’s client: VSCode/Jupyter (interactive notebooks)
- Code modules:
– Data ingestion (credit/data.py, stock_prices/data_manager.py)
– Experiment scripts (exp__*.py)
– Utilities (logger.py)
- ML pipeline stages: data → features → training → tracking → explainability
- Experiment-tracking backend (Docker Compose):
– MLflow Tracking Server
– PostgreSQL metadata store
– MinIO artifact store
- Instructional content (slides, assignments) – static

b. Relationships and data flows
1. Learner opens a Jupyter notebook → invokes data ingestion modules → yields raw data
2. Notebook or script runs pipeline/experiments → calls MLflow Python client
3. MLflow client writes metrics to Tracking Server
5. Artifacts (models, plots) pushed to MinIO buckets
6. Student reads results back in notebook for explainability and monitoring demos

c. Architectural patterns & design principles
- Modularity: separate folders for content, code demos, infra
- Infrastructure as code: docker-compose defines a small microservice ecosystem
- Clear separation of concerns:
– Data ingestion vs. model training vs. logging/tracking
- Reproducibility: Environment specs (environment.yaml, .env) and containerization

d. Key technologies & frameworks
- Python + Jupyter Notebooks
- MLflow (experiment tracking)
- MinIO (S3-compatible object storage)
- PostgreSQL (tracking DB)
- Docker & Docker Compose
- VSCode (Jupyter + Python extensions)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Educational Content
    subgraph "Educational Content"
        direction TB
        N1["Live-Lab Notebooks & PDF Slides"]:::content
        N2["Graded Assignments & Rubrics"]:::content
        MD1["Markdown Slides"]:::content
        MD2["Slide Generator Script"]:::content
        N4["Cohort-Specific Extras"]:::content
    end

    %% Demo Code Modules
    subgraph "Demo Code (05_src)"
        direction TB
        DM1["Data Ingestion (credit/data.py)"]:::code
        DM2["Data Manager (stock_prices/data_manager.py)"]:::code
        U1["Logger Utility (utils/logger.py)"]:::code
        subgraph "Experiment Scripts"
            direction TB
            ES1["exp__logistic_simple.py"]:::code
            ES2["exp__logistic_grid_search.py"]:::code
            ES3["exp__logistic_hyperopt.py"]:::code
        end
    end

    %% Pipeline & Client
    MLC["MLflow Client"]:::code

    %% Docker Compose Stack
    subgraph "Docker Compose Stack"
        direction TB
        DC["docker-compose.yml"]:::infra
        ML["MLflow Tracking Server"]:::infra
        PG["PostgreSQL Metadata Store"]:::infra
        MI["MinIO Artifact Store"]:::infra

        subgraph "Experiment Tracking Configs"
            direction TB
            MLcfg["MLflow Dockerfile & requirements.txt"]:::infra
            MIcfg["MinIO create-bucket.sh"]:::infra
            PGcfg["PostgreSQL init.sql"]:::infra
            TS["MLflow Integration Test Script"]:::infra
        end
    end

    %% Configuration Files
    subgraph "Configurations"
        direction TB
        E1["Root .env / environment.yaml"]:::infra
        E2["05_src/.env"]:::infra
        E3["05_src/experiment_tracking/.env"]:::infra
    end

    %% Data Flow
    N1 -->|opens notebook| DM1
    N1 -->|opens notebook| DM2
    DM1 --> ES1
    DM1 --> ES2
    DM1 --> ES3
    DM2 --> ES1
    DM2 --> ES2
    DM2 --> ES3
    DM1 --> U1
    DM2 --> U1
    ES1 --> U1
    ES2 --> U1
    ES3 --> U1
    ES1 -->|runs experiment| MLC
    ES2 -->|runs experiment| MLC
    ES3 -->|runs experiment| MLC
    MLC -->|API calls| DC
    DC --> ML
    DC --> PG
    DC --> MI
    DC --> MLcfg
    DC --> MIcfg
    DC --> PGcfg
    DC --> TS
    ML -->|stores metadata| PG
    ML -->|stores artifacts| MI
    PG -->|query results| N1
    MI -->|fetch artifacts| N1

    E1 --> DC
    E2 --> DC
    E3 --> DC

    %% Click Events
    click N1 "https://github.com/vincentschaik/production/tree/main/01_materials"
    click N2 "https://github.com/vincentschaik/production/tree/main/02_activities"
    click MD1 "https://github.com/vincentschaik/production/tree/main/03_instructional_team/markdown_slides"
    click MD2 "https://github.com/vincentschaik/production/blob/main/03_instructional_team/generate_slides.sh"
    click N4 "https://github.com/vincentschaik/production/tree/main/04_this_cohort"
    click DM1 "https://github.com/vincentschaik/production/blob/main/05_src/credit/data.py"
    click ES1 "https://github.com/vincentschaik/production/blob/main/05_src/credit/exp__logistic_simple.py"
    click ES2 "https://github.com/vincentschaik/production/blob/main/05_src/credit/exp__logistic_grid_search.py"
    click ES3 "https://github.com/vincentschaik/production/blob/main/05_src/credit/exp__logistic_hyperopt.py"
    click DM2 "https://github.com/vincentschaik/production/blob/main/05_src/stock_prices/data_manager.py"
    click U1 "https://github.com/vincentschaik/production/blob/main/05_src/utils/logger.py"
    click DC "https://github.com/vincentschaik/production/blob/main/05_src/experiment_tracking/docker-compose.yml"
    click MLcfg "https://github.com/vincentschaik/production/tree/main/05_src/experiment_tracking/mlflow/Dockerfile"
    click MIcfg "https://github.com/vincentschaik/production/blob/main/05_src/experiment_tracking/minio/create-bucket.sh"
    click PGcfg "https://github.com/vincentschaik/production/blob/main/05_src/experiment_tracking/postgres/init.sql"
    click TS "https://github.com/vincentschaik/production/blob/main/05_src/experiment_tracking/test_mlflow.py"

    %% Styles
    classDef content fill:#D6EAF8,stroke:#1B4F72,color:#154360
    classDef code fill:#D5F5E3,stroke:#196F3D,color:#145A32
    classDef infra fill:#FDEBD0,stroke:#873600,color:#6E2C00
```
