How feifeiwww/20220726_databricks_demo_transfer_learning_with_mlflow works
Project type and purpose
• This is a hands-on demo repository for an image classification workflow (Cats vs Dogs) using transfer learning on Databricks, with MLflow to track experiments. It’s not a production service but an interactive notebook-based data science pipeline.
File structure
• Top-level files:
· README.md – instructions and overview.
· slides/FeifeiFourthBrainPresentation.pdf – slide deck.
· notebooks/Transfer Learning Demo Part 1.ipynb
· notebooks/Transfer Learning Demo Part 2.ipynb
• There’s no backend service code; everything runs in Databricks notebooks on a cluster.
Extract architecture hints from the README
• Execution environment: Databricks Community Edition cluster (Runtime 10.4 LTS ML).
• Core libraries: TensorFlow for transfer learning, MLflow for experiment tracking (and optional Model Registry).
• Data: users import/download notebook, dataset loaded in notebook (likely from public URL or Databricks FileStore).
• User interacts via Databricks UI; experiments are viewed through MLflow’s Experiments UI.
a. Main components
1. User’s browser & Databricks Workspace UI
2. Databricks Compute Cluster (driver + executors)
3. Notebooks (Part 1 and Part 2)
4. Data storage (DBFS or external storage mounting the Cats vs Dogs dataset)
5. Transfer Learning framework (TensorFlow pre-trained model)
6. MLflow Tracking Server (Databricks-managed)
7. MLflow UI and optional Model Registry
b. Relationships and data flow
1. User opens notebook through Databricks Workspace (UI).
2. Notebook code executes on the Compute Cluster.
3. Cluster nodes read training data from Data Storage.
4. TensorFlow on the cluster loads a pre-trained model, fine-tunes on the dataset.
5. During training, notebook calls MLflow APIs to log metrics, parameters, artifacts to the MLflow Tracking Server.
6. User inspects experiment runs in the MLflow UI (via Workspace).
7. (Optional in Part 2) The trained model is registered in the MLflow Model Registry.
c. Key patterns and design principles
• Interactive notebook workflow – code, visualizations, and narrative live together.
• Separation of concerns: compute (cluster) vs. storage (DBFS/S3) vs. tracking (MLflow server).
• Modular pipeline: data ingestion → transfer learning training → experiment logging → model registry.
d. Technologies & frameworks
• Databricks Runtime 10.4 ML (Spark + pre-installed TensorFlow & MLflow)
• Jupyter/Databricks notebooks
• TensorFlow (Keras) for model fine-tuning
• MLflow for experiment tracking & Model Registry
• Databricks FileStore or external storage (S3/ADLS) for data