# end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35 architecture

> Architecture diagram of the GitHub repository end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35
- Repository: https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35

## Overview

##### Project type and purpose
- This repo is not a web app or microservice, but a “course” repository for an end-to-end MLOps curriculum using Databricks.
- It contains:
- A Python package (`src/mlops_course`)
- Task definitions (Taskfile.yml / Taskfiles.md) for local environment orchestration
- CI/CD workflows (`.github/workflows`) to validate, test, and publish the package

##### File structure
- Top level
- .github/workflows — CI (ci.yml) and CD (cd.yml) pipelines
- Taskfile.yml & Taskfiles.md — defines commands such as `task sync-dev`
- pyproject.toml & version.txt — Python package metadata and versioning
- src/mlops_course — the actual Python code for the course
- Tooling areas
- .pre-commit-config.yaml — linting/formatting checks
- .gitignore — excludes venv, lockfiles, etc.
- No frontend folder—this is purely a backend/library‐style project with DevOps emphasis

##### Key insights from the README
- Environment setup targets Python 3.11 on Databricks 15.4 LTS
- Uses UV (astral.sh/uv) for venv management
- Uses Taskfile as a task runner to orchestrate:
- Virtual environment creation
- Dependency sync (`task sync-dev`)
- Running Databricks jobs or notebooks
- Workflow: developer clones → sets GIT_TOKEN → runs `task sync-dev` → activates venv → develops & tests locally or on Databricks → opens PR → CI runs → merge → CD publishes

a. Main components
- Developer Workstation (Local Env)
- Task Runner & Virtual Env (UV + Taskfile + .venv)
- Source Code Repository (GitHub)
- CI Pipeline (GitHub Actions ci.yml)
- CD Pipeline (GitHub Actions cd.yml)
- Databricks Workspace/Cluster (Compute)
- Artifact Registry (PyPI or GitHub Packages)
- External Dataset (student’s own data)

b. Relationships & interactions
1. Developer writes code against `src/mlops_course` and defines tasks in Taskfile.yml
2. Developer runs `task sync-dev` locally → UV sets up .venv → dependencies installed
3. Code is tested locally or via Databricks API calls
4. Push to GitHub → PR opened → CI pipeline triggers
- Linting, formatting (pre-commit)
- Unit tests
- Version consistency check against version.txt
5. On merge to main → CD pipeline triggers
- Builds wheel / sdist via pyproject.toml
- Publishes to artifact registry (PyPI/GitHub Packages)
6. Developer (or course scripts) uses published package or source to run notebooks/jobs on Databricks

c. Architectural patterns & principles
- Clear separation of concerns:
- Local dev vs. cloud compute
- Build/test pipeline vs. release pipeline
- Use of a task runner (Taskfile) to enforce reproducible workflows
- Version-driven packaging via version.txt + pyproject.toml
- GitOps approach: all changes via PRs, CI gating merges

d. Core technologies & libraries
- Python 3.11
- UV for virtualenv management
- Taskfile for task orchestration
- GitHub Actions for CI/CD
- Databricks REST/API for remote compute
- pre-commit for linting/formatting

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local Development Zone
    subgraph "Local Development" 
        direction TB
        DevWorkstation["Developer Workstation"]:::local
        UV["UV / .venv"]:::local
        TaskDefs["Taskfile.yml & Taskfiles.md"]:::local
        PreCommit[".pre-commit-config.yaml & .gitignore"]:::local
        PackageSrc["src/mlops_course"]:::local
        InitPy["__init__.py"]:::local
        PyProject["pyproject.toml"]:::local
        VersionTxt["version.txt"]:::local
    end

    %% GitHub Zone
    subgraph "GitHub & CI/CD" 
        direction TB
        Repo["Source Code Repository"]:::git
        CI["CI Pipeline (ci.yml)"]:::git
        CD["CD Pipeline (cd.yml)"]:::git
        CodeOwners[".github/CODEOWNERS"]:::git
        README["README.md"]:::git
    end

    %% Cloud Zone
    subgraph "Databricks" 
        direction TB
        DBWorkspace["Databricks Workspace/Cluster"]:::databricks
    end

    %% External Systems
    subgraph "External" 
        direction TB
        ArtifactReg["Artifact Registry (PyPI/GH Packages)"]:::external
        ExtData["External Dataset"]:::external
    end

    %% Local Flow
    DevWorkstation -->|writes code & tasks| PackageSrc
    PackageSrc -->|defines entry| InitPy
    DevWorkstation -->|edits metadata| PyProject
    DevWorkstation -->|updates version| VersionTxt
    DevWorkstation -->|configures tasks| TaskDefs
    DevWorkstation -->|runs lint & format| PreCommit
    DevWorkstation -->|runs task sync-dev| UV
    UV -->|installs dependencies| PackageSrc

    %% Push to GitHub
    DevWorkstation -->|push code| Repo
    TaskDefs -->|push tasks| Repo
    PreCommit -->|pre-commit config| Repo
    PyProject -->|package metadata| Repo
    VersionTxt -->|version file| Repo

    %% CI/CD Flow
    Repo -->|on PR| CI
    CI -->|lint, test, version check| Repo
    Repo -->|on merge| CD
    CD -->|build & publish| ArtifactReg

    %% Databricks Jobs
    DevWorkstation -->|invoke notebooks/jobs| DBWorkspace
    DBWorkspace -->|runs with package| ArtifactReg
    DBWorkspace -->|access data| ExtData

    %% Click Events
    click PackageSrc "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/tree/main/src/mlops_course/"
    click InitPy "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/src/mlops_course/__init__.py"
    click PyProject "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/pyproject.toml"
    click VersionTxt "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/version.txt"
    click TaskDefs "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/Taskfile.yml"
    click TaskDefs "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/Taskfiles.md"
    click PreCommit "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/.pre-commit-config.yaml"
    click PreCommit "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/.gitignore"
    click CI "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/.github/workflows/ci.yml"
    click CD "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/.github/workflows/cd.yml"
    click CodeOwners "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/tree/main/.github/CODEOWNERS"
    click README "https://github.com/end-to-end-mlops-databricks-3/marvelous-databricks-course-malganis35/blob/main/README.md"

    %% Styles
    classDef local fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef git fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef databricks fill:#d4edda,stroke:#333,stroke-width:1px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:1px
```
