# feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow architecture

> Architecture diagram of the GitHub repository feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow
- Repository: https://github.com/feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow

## Overview

##### Project type and purpose
- This is a hands-on demo repository for an image classification workflow (Cats vs Dogs) using transfer learning on Databricks, with MLflow to track experiments. It’s not a production service but an interactive notebook-based data science pipeline.

##### File structure
- Top-level files:
· README.md – instructions and overview.
· slides/FeifeiFourthBrainPresentation.pdf – slide deck.
· notebooks/Transfer Learning Demo Part 1.ipynb
· notebooks/Transfer Learning Demo Part 2.ipynb
- There’s no backend service code; everything runs in Databricks notebooks on a cluster.

##### Extract architecture hints from the README
- Execution environment: Databricks Community Edition cluster (Runtime 10.4 LTS ML).
- Core libraries: TensorFlow for transfer learning, MLflow for experiment tracking (and optional Model Registry).
- Data: users import/download notebook, dataset loaded in notebook (likely from public URL or Databricks FileStore).
- User interacts via Databricks UI; experiments are viewed through MLflow’s Experiments UI.

a. Main components
1. User’s browser & Databricks Workspace UI
2. Databricks Compute Cluster (driver + executors)
3. Notebooks (Part 1 and Part 2)
4. Data storage (DBFS or external storage mounting the Cats vs Dogs dataset)
5. Transfer Learning framework (TensorFlow pre-trained model)
6. MLflow Tracking Server (Databricks-managed)
7. MLflow UI and optional Model Registry

b. Relationships and data flow
1. User opens notebook through Databricks Workspace (UI).
2. Notebook code executes on the Compute Cluster.
3. Cluster nodes read training data from Data Storage.
4. TensorFlow on the cluster loads a pre-trained model, fine-tunes on the dataset.
5. During training, notebook calls MLflow APIs to log metrics, parameters, artifacts to the MLflow Tracking Server.
6. User inspects experiment runs in the MLflow UI (via Workspace).
7. (Optional in Part 2) The trained model is registered in the MLflow Model Registry.

c. Key patterns and design principles
- Interactive notebook workflow – code, visualizations, and narrative live together.
- Separation of concerns: compute (cluster) vs. storage (DBFS/S3) vs. tracking (MLflow server).
- Modular pipeline: data ingestion → transfer learning training → experiment logging → model registry.

d. Technologies & frameworks
- Databricks Runtime 10.4 ML (Spark + pre-installed TensorFlow & MLflow)
- Jupyter/Databricks notebooks
- TensorFlow (Keras) for model fine-tuning
- MLflow for experiment tracking & Model Registry
- Databricks FileStore or external storage (S3/ADLS) for data

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Project Files"
        README["README.md"]:::docs
        Note1["Transfer Learning Demo Part 1.ipynb"]:::docs
        Note2["Transfer Learning Demo Part 2.ipynb"]:::docs
        Slides["FeifeiFourthBrainPresentation.pdf"]:::docs
    end

    subgraph "Presentation/UI"
        User["User Browser"]:::ui
        Workspace["Databricks Workspace UI"]:::ui
        MLflowUI["MLflow UI"]:::ui
    end

    subgraph "Compute"
        Cluster["Databricks Compute Cluster\n(TensorFlow & MLflow Client)"]:::compute
    end

    subgraph "Storage/Tracking"
        DataStore["Data Storage\n(DBFS/S3)"]:::storage
        Tracking["MLflow Tracking Server"]:::storage
        Registry["MLflow Model Registry"]:::storage
    end

    User -->|"opens notebooks"| Workspace
    Workspace -->|"executes code"| Cluster
    Cluster -->|"reads training data"| DataStore
    Cluster -->|"logs metrics, parameters, artifacts"| Tracking
    Cluster -->|"registers trained model"| Registry
    User -->|"views experiment runs"| MLflowUI
    MLflowUI -->|"retrieves logs"| Tracking

    click README "https://github.com/feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow/blob/main/README.md"
    click Note1 "https://github.com/feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow/blob/main/notebooks/Transfer Learning Demo Part 1.ipynb"
    click Note2 "https://github.com/feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow/blob/main/notebooks/Transfer Learning Demo Part 2.ipynb"
    click Slides "https://github.com/feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow/blob/main/slides/FeifeiFourthBrainPresentation.pdf"

    classDef ui fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef compute fill:#00BFFF,stroke:#333,stroke-width:1px;
    classDef storage fill:#32CD32,stroke:#333,stroke-width:1px;
    classDef docs fill:#D3D3D3,stroke:#333,stroke-width:1px;
```
