# kaggle/docker-python architecture

> Architecture diagram of the GitHub repository kaggle/docker-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaggle/docker-python
- Repository: https://github.com/kaggle/docker-python

## Overview

##### Project type and purpose
- This is an open-source build and distribution tool for Kaggle’s Python Docker images.
- It defines, builds, tests, and publishes CPU, GPU, and TPU variants of the “python” container used by Kaggle Notebooks.

##### High-level architecture components
- Source repository: contains Dockerfile templates, build scripts, patch definitions, package-specific Dockerfiles, tests, and CI config
- Build orchestrator: shell scripts (`build`, `clean-layer.sh`, specialized Dockerfiles under `dev.Dockerfile`, `Dockerfile.tmpl`, `tpu/`) that assemble base images and layer on requested Python packages and patches
- Package build definitions: under `packages/` (e.g., `torch.Dockerfile`, `jaxlib.Dockerfile`) and `patches/` (module shims and security configs)
- Test harness: `test` script and the comprehensive test suite under `tests/` that spins up built images and runs package-compatibility checks
- CI/CD pipeline: `Jenkinsfile` and GitHub issue/pr templates automate image builds, tests, and publishing
- Artifact registry: Google Container Registry (GCR) hosts the final images under `gcr.io/kaggle-images/python` and `gcr.io/kaggle-gpu-images/python`

##### Relationships and data flows
a. Developer edits `kaggle_requirements.txt` or adds new `packages/…` and patches under `patches/`.
b. `build` script picks up `Dockerfile.tmpl`, applies `config.txt`, merges `packages/` layers and `patches/`, emits CPU and/or GPU Dockerfiles.
c. Docker daemon builds the images (via `dev.Dockerfile`, `Dockerfile.tmpl`, and `tpu/Dockerfile` for TPU variant).
d. `test` script launches containers from the newly built images and runs tests in `tests/`; failures feed back to developer.
e. CI (Jenkins/GitHub Actions) triggers the build & test steps on PR; on success, images are pushed to GCR.
f. Kaggle Notebooks runtime pulls published images from GCR at notebook‐launch time.

##### Key architectural patterns and design principles
- Layered Docker image strategy: base image → system-level dependencies → Python packages → patches → final image
- Modular package definitions: each library or framework has its own Dockerfile under `packages/`, making it easy to update or add support
- Centralized patch directory to override or extend behavior (e.g., custom `kaggle_web_client.py`, notebook extensions)
- CI/CD as code: the `Jenkinsfile` and `.github` templates formalize the delivery pipeline

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Repository
    SourceRepo["Source Repository"]:::code
    click SourceRepo "https://github.com/kaggle/docker-python/tree/main//"

    %% Build Orchestrator
    subgraph "Build Orchestrator"
        direction TB
        BuildScript["build script (./build)"]:::config
        click BuildScript "https://github.com/kaggle/docker-python/tree/main//build"
        CleanLayer["clean-layer.sh"]:::config
        click CleanLayer "https://github.com/kaggle/docker-python/blob/main//clean-layer.sh"
        DockerfileTmpl["Dockerfile.tmpl"]:::config
        click DockerfileTmpl "https://github.com/kaggle/docker-python/blob/main//Dockerfile.tmpl"
        DevDocker["dev.Dockerfile"]:::config
        click DevDocker "https://github.com/kaggle/docker-python/blob/main//dev.Dockerfile"
        ConfigTxt["config.txt"]:::config
        click ConfigTxt "https://github.com/kaggle/docker-python/blob/main//config.txt"
        TPUFolder["tpu/"]:::config
        click TPUFolder "https://github.com/kaggle/docker-python/tree/main//tpu"
    end

    %% Package Definitions and Patches
    PackageDefs["Package Definitions (/packages)"]:::build
    click PackageDefs "https://github.com/kaggle/docker-python/tree/main//packages"
    Patches["Patches (/patches)"]:::build
    click Patches "https://github.com/kaggle/docker-python/tree/main//patches"

    %% Docker Daemon
    DockerDaemon["Docker Daemon"]:::build

    %% Test Harness
    subgraph "Test Harness"
        direction TB
        TestScript["test script (/test)"]:::test
        click TestScript "https://github.com/kaggle/docker-python/tree/main//test"
        TestSuite["Test Suite (/tests)"]:::test
        click TestSuite "https://github.com/kaggle/docker-python/tree/main//tests"
    end

    %% CI/CD Pipeline
    subgraph "CI/CD Pipeline"
        direction TB
        JenkinsFile["Jenkinsfile"]:::test
        click JenkinsFile "https://github.com/kaggle/docker-python/tree/main//Jenkinsfile"
        GitHubTemplates[".github/ISSUE_TEMPLATE"]:::test
        click GitHubTemplates "https://github.com/kaggle/docker-python/tree/main//.github/ISSUE_TEMPLATE"
    end

    %% Utilities & Helpers
    DiffUtil["Docker diff utility (/diff)"]:::config
    click DiffUtil "https://github.com/kaggle/docker-python/tree/main//diff"
    PushHelper["Artifact push helper (/push)"]:::config
    click PushHelper "https://github.com/kaggle/docker-python/tree/main//push"

    %% External Services
    GCR["GCR Registry"]:::external
    KaggleRuntime["Kaggle Notebooks Runtime"]:::external

    %% Flows
    SourceRepo --> BuildScript
    SourceRepo --> CleanLayer
    SourceRepo --> DockerfileTmpl
    SourceRepo --> DevDocker
    SourceRepo --> ConfigTxt
    SourceRepo --> TPUFolder
    SourceRepo --> PackageDefs
    SourceRepo --> Patches

    PackageDefs --> BuildScript
    Patches --> BuildScript

    BuildScript --> DockerDaemon
    CleanLayer --> DockerDaemon
    DevDocker --> DockerDaemon
    DockerfileTmpl --> DockerDaemon
    TPUFolder --> DockerDaemon

    DockerDaemon --> TestScript
    DockerDaemon --> TestSuite

    TestScript --> JenkinsFile
    TestSuite --> JenkinsFile

    JenkinsFile --> GCR
    GitHubTemplates --> GCR

    GCR --> KaggleRuntime

    %% Styles
    classDef build fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef test fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px
    class SourceRepo,BuildScript,CleanLayer,DockerfileTmpl,DevDocker,ConfigTxt,TPUFolder build
    class DockerDaemon,PackageDefs,Patches build
    class TestScript,TestSuite,JenkinsFile,GitHubTemplates test
    class GCR,KaggleRuntime external
    class DiffUtil,PushHelper config
```
