# davidadsp/generative_deep_learning_2nd_edition architecture

> Architecture diagram of the GitHub repository davidadsp/generative_deep_learning_2nd_edition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/davidadsp/generative_deep_learning_2nd_edition
- Repository: https://github.com/davidadsp/generative_deep_learning_2nd_edition

## Overview

#### Purpose and Entry Points
This is the companion codebase for *Generative Deep Learning, 2nd Edition*: an interactive, notebook-first collection of model implementations and exercises. Users primarily start Docker Compose, open Jupyter, and execute chapter/example notebooks under `notebooks/`. Runtime setup is defined by `docker-compose.yml`, GPU alternative `docker-compose.gpu.yml`, and image definitions in `docker/Dockerfile` and `docker/Dockerfile.gpu`.

#### Execution Environment
Docker provides the reproducible Python/Jupyter environment, with CPU and GPU deployment variants. Environment variables from a local `.env` (templated by `sample.env`) supply Jupyter/TensorBoard ports and Kaggle credentials. Cloud GPU usage is documented in `docs/googlecloud.md`; local container operation in `docs/docker.md`. Dependencies are centralized in `requirements.txt`.

#### Notebook Curriculum as Application Surface
Each notebook is a self-contained training/inference workflow rather than part of a shared application service. The content progresses from neural-network foundations through generative-model families:
- MLP/CNN foundations: `notebooks/02_deeplearning/01_mlp/mlp.ipynb`, `notebooks/02_deeplearning/02_cnn/cnn.ipynb`
- VAEs: `notebooks/03_vae/01_autoencoder/autoencoder.ipynb`, `notebooks/03_vae/02_vae_fashion/vae_fashion.ipynb`
- GANs: `notebooks/04_gan/01_dcgan/dcgan.ipynb`, `notebooks/04_gan/02_wgan_gp/wgan_gp.ipynb`
- Autoregressive models: `notebooks/05_autoregressive/01_lstm/lstm.ipynb`, `notebooks/05_autoregressive/02_pixelcnn/pixelcnn.ipynb`
- Later methods and applications: `notebooks/06_normflow/01_realnvp/realnvp.ipynb`, `notebooks/08_diffusion/01_ddm/ddm.ipynb`, `notebooks/09_transformer/gpt/gpt.ipynb`.

#### Training Artifact Boundary
Examples conventionally separate generated state from source: each model directory has `checkpoint/`, `logs/`, `models/`, and `output/` directories, generally gitignored. Notebook execution reads prepared/downloaded data, trains models, writes recoverable checkpoints and saved models, emits TensorBoard logs, and produces samples/visualizations. Representative layout: `notebooks/04_gan/01_dcgan/checkpoint`, `notebooks/04_gan/01_dcgan/logs`, `notebooks/04_gan/01_dcgan/output`.

#### Shared and Domain-Specific Python Support
Most implementation logic lives inline in notebooks; reusable support is intentionally limited. Cross-notebook utilities reside in `notebooks/utils.py`. Complex domain-specific preprocessing/model support is colocated with music notebooks: `notebooks/11_music/01_transformer/transformer_utils.py` and `notebooks/11_music/02_musegan/musegan_utils.py`. VAE face-specific support is in `notebooks/03_vae/03_vae_faces/vae_utils.py`.

#### Data Acquisition and External Services
Persistent datasets are expected beneath `data/`, excluded from version control. `scripts/download.sh` is the data-ingestion entry point and dispatches source-specific downloaders, including Kaggle-backed acquisition in `scripts/downloaders/download_kaggle_data.sh` and music-data downloaders such as `scripts/downloaders/download_bach_chorale_data.sh`. Kaggle is the material external service and requires `KAGGLE_USERNAME`/`KAGGLE_KEY`; downloaded data forms the boundary between external sources and local notebook workflows.

#### Observability
TensorBoard is the experiment-monitoring surface. `scripts/tensorboard.sh` targets a selected chapter/example log directory and exposes it through the configured host port. This couples directly to per-example `logs/` outputs, for example `notebooks/08_diffusion/01_ddm/logs` and `notebooks/11_music/01_transformer/logs`.

Main entry points: [`docker-compose.yml`](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/docker-compose.yml) (Docker Compose), [`scripts/download.sh`](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/download.sh) (Data Download Entry Point), [`docker/Dockerfile`](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/docker/Dockerfile) (Jupyter Images), [`notebooks/04_gan/01_dcgan/output`](https://github.com/davidadsp/generative_deep_learning_2nd_edition/tree/main/notebooks/04_gan/01_dcgan/output) (Per-Example Outputs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_environment["Execution Environment"]
  node_compose{{"Docker Compose<br/>runtime orchestration<br/>[docker-compose.yml]"}}
  node_images["Jupyter Images<br/>Docker images"]
  node_dependencies["Python Dependencies<br/>dependency manifest<br/>[requirements.txt]"]
  node_jupyter["Jupyter Notebook Runtime<br/>interactive runtime"]
end

subgraph group_curriculum["Notebook Curriculum"]
  node_utilities["Shared Notebook Utilities<br/>Python support<br/>[utils.py]"]
  node_foundations["MLP and CNN<br/>foundation notebooks<br/>[mlp.ipynb]"]
  node_vae["VAE Workflows<br/>generative notebooks<br/>[vae_fashion.ipynb]"]
  node_gan["GAN Workflows<br/>generative notebooks<br/>[dcgan.ipynb]"]
  node_autoregressive["Autoregressive Workflows<br/>generative notebooks<br/>[lstm.ipynb]"]
  node_later_models["Flows, Diffusion, Transformers<br/>advanced notebooks<br/>[ddm.ipynb]"]
  node_music_support["Music Model Support<br/>domain Python support"]
end

subgraph group_data["Data Acquisition"]
  node_download["Data Download Entry Point<br/>ingestion script<br/>[download.sh]"]
  node_kaggle(("Kaggle<br/>external dataset service"))
  node_datasets[("Local Datasets<br/>persistent data")]
end

subgraph group_artifacts["Training Artifacts"]
  node_training_outputs[("Per-Example Outputs<br/>generated artifacts")]
  node_tensorboard_script["TensorBoard Launcher<br/>monitoring script<br/>[tensorboard.sh]"]
  node_tensorboard["TensorBoard<br/>experiment monitoring"]
end

node_compose -->|"builds/runs"| node_images
node_images -->|"installs"| node_dependencies
node_compose -->|"starts"| node_jupyter
node_jupyter -->|"executes"| node_foundations
node_jupyter -->|"executes"| node_vae
node_jupyter -->|"executes"| node_gan
node_jupyter -->|"executes"| node_autoregressive
node_jupyter -->|"executes"| node_later_models
node_jupyter -.->|"imports"| node_utilities
node_jupyter -.->|"uses for music examples"| node_music_support
node_download -->|"acquires datasets from"| node_kaggle
node_download -->|"writes"| node_datasets
node_datasets -->|"feeds workflows"| node_jupyter
node_jupyter -->|"writes checkpoints, models, logs, samples"| node_training_outputs
node_training_outputs -->|"log directory"| node_tensorboard_script
node_tensorboard_script -->|"launches"| node_tensorboard

click node_compose "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/docker-compose.yml"
click node_images "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/docker/Dockerfile"
click node_dependencies "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/requirements.txt"
click node_utilities "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/utils.py"
click node_foundations "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/02_deeplearning/01_mlp/mlp.ipynb"
click node_vae "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/03_vae/02_vae_fashion/vae_fashion.ipynb"
click node_gan "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/04_gan/01_dcgan/dcgan.ipynb"
click node_autoregressive "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/05_autoregressive/01_lstm/lstm.ipynb"
click node_later_models "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/08_diffusion/01_ddm/ddm.ipynb"
click node_music_support "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/11_music/01_transformer/transformer_utils.py"
click node_download "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/download.sh"
click node_kaggle "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/downloaders/download_kaggle_data.sh"
click node_training_outputs "https://github.com/davidadsp/generative_deep_learning_2nd_edition/tree/main/notebooks/04_gan/01_dcgan/output"
click node_tensorboard_script "https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/tensorboard.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_compose,node_images,node_dependencies,node_jupyter toneBlue
class node_utilities,node_foundations,node_vae,node_gan,node_autoregressive,node_later_models,node_music_support toneAmber
class node_download,node_kaggle,node_datasets toneMint
class node_training_outputs,node_tensorboard_script,node_tensorboard toneRose
```

## Components

### Execution Environment

- [Docker Compose](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/docker-compose.yml): runtime orchestration · CPU and GPU Compose variants launch the notebook environment. · `docker-compose.yml`
- [Jupyter Images](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/docker/Dockerfile): Docker images · CPU and GPU image definitions provide the reproducible Python runtime. · `docker/Dockerfile`
- [Python Dependencies](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/requirements.txt): dependency manifest · `requirements.txt`
- **Jupyter Notebook Runtime**: interactive runtime · Primary user-facing execution environment.

### Notebook Curriculum

- [Shared Notebook Utilities](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/utils.py): Python support · `notebooks/utils.py`
- [MLP and CNN](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/02_deeplearning/01_mlp/mlp.ipynb): foundation notebooks · `notebooks/02_deeplearning/01_mlp/mlp.ipynb`
- [VAE Workflows](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/03_vae/02_vae_fashion/vae_fashion.ipynb): generative notebooks · Autoencoder, Fashion VAE, and face VAE examples. · `notebooks/03_vae/02_vae_fashion/vae_fashion.ipynb`
- [GAN Workflows](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/04_gan/01_dcgan/dcgan.ipynb): generative notebooks · DCGAN and WGAN-GP training examples. · `notebooks/04_gan/01_dcgan/dcgan.ipynb`
- [Autoregressive Workflows](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/05_autoregressive/01_lstm/lstm.ipynb): generative notebooks · LSTM and PixelCNN examples. · `notebooks/05_autoregressive/01_lstm/lstm.ipynb`
- [Flows, Diffusion, Transformers](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/08_diffusion/01_ddm/ddm.ipynb): advanced notebooks · Later generative-model workflows. · `notebooks/08_diffusion/01_ddm/ddm.ipynb`
- [Music Model Support](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/notebooks/11_music/01_transformer/transformer_utils.py): domain Python support · Colocated preprocessing and model utilities for music notebooks. · `notebooks/11_music/01_transformer/transformer_utils.py`

### Data Acquisition

- [Data Download Entry Point](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/download.sh): ingestion script · Dispatches source-specific dataset downloaders. · `scripts/download.sh`
- [Kaggle](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/downloaders/download_kaggle_data.sh): external dataset service · Credentials are supplied through the local environment. · `scripts/downloaders/download_kaggle_data.sh`
- **Local Datasets**: persistent data · Downloaded and prepared notebook input data.

### Training Artifacts

- [Per-Example Outputs](https://github.com/davidadsp/generative_deep_learning_2nd_edition/tree/main/notebooks/04_gan/01_dcgan/output): generated artifacts · Checkpoints, saved models, logs, and samples are kept beside each example. · `notebooks/04_gan/01_dcgan/output`
- [TensorBoard Launcher](https://github.com/davidadsp/generative_deep_learning_2nd_edition/blob/main/scripts/tensorboard.sh): monitoring script · Selects an example log directory and exposes TensorBoard. · `scripts/tensorboard.sh`
- **TensorBoard**: experiment monitoring

## Connections

- Docker Compose → Jupyter Images: builds/runs
- Jupyter Images → Python Dependencies: installs
- Docker Compose → Jupyter Notebook Runtime: starts
- Jupyter Notebook Runtime → MLP and CNN: executes
- Jupyter Notebook Runtime → VAE Workflows: executes
- Jupyter Notebook Runtime → GAN Workflows: executes
- Jupyter Notebook Runtime → Autoregressive Workflows: executes
- Jupyter Notebook Runtime → Flows, Diffusion, Transformers: executes
- Jupyter Notebook Runtime → Shared Notebook Utilities: imports
- Jupyter Notebook Runtime → Music Model Support: uses for music examples
- Data Download Entry Point → Kaggle: acquires datasets from
- Data Download Entry Point → Local Datasets: writes
- Local Datasets → Jupyter Notebook Runtime: feeds workflows
- Jupyter Notebook Runtime → Per-Example Outputs: writes checkpoints, models, logs, samples
- Per-Example Outputs → TensorBoard Launcher: log directory
- TensorBoard Launcher → TensorBoard: launches