# google-deepmind/deepmind-research architecture

> Architecture diagram of the GitHub repository google-deepmind/deepmind-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/deepmind-research
- Repository: https://github.com/google-deepmind/deepmind-research

## Overview

##### Project type and purpose
- This is a monolithic “monorepo” hosting dozens of independent research projects and accompanying code, notebooks, data, and scripts.
- Each top‐level folder (e.g., `adversarial_robustness`, `avae`, `fusion_tcv`, `wikigraphs`, …) is a self‐contained research package illustrating DeepMind work in a particular paper.
- The root `README.md`, `.travis.yml`, `LICENSE`, `CONTRIBUTING.md` describe the umbrella repository and its CI/CD.

##### File structure
- Top level:
- Monorepo root (contains global CI config and docs)
- One folder per project (each a Python package or demo)
- Within each project folder you typically find:
- Source code modules (`*.py`, sometimes subpackages)
- Jupyter notebooks (`*.ipynb`) for demos/analysis
- Data or checkpoint assets (e.g. `*.npz`, TFRecords)
- Entry scripts (`run.sh`, `main.py`, experiment drivers)
- Packaging support (`requirements.txt`, `setup.py`, `BUILD.bazel`)
- Unit tests (`*_test.py`)
- There is no cross‐project shared library – each project depends only on its own declared requirements.

##### README for additional insights
- The root README lists ~60 projects, each tied to a publication.
- Many use common ML frameworks (JAX, PyTorch, TensorFlow) but keep their codebases isolated.
- Some use Bazel (e.g. `density_functional_approximation_dm21`), most use standard Python packaging.

a. Main components
1. Monorepo Root (“DeepMind Research”)
2. CI/CD Pipeline (TravisCI)
3. Developer Environment (local or cloud dev machine, conda/venv)
4. Individual Project Packages (one per paper)
– Source code
– Notebooks
– Data assets
– Scripts & tests
5. Dependency Manager & Build Tools (pip, Bazel)
6. External Hosts/Services (GitHub, TFHub, external datasets)

b. Relationships & interactions
– Developer clones monorepo → creates virtual environment → installs global tools (e.g. Bazel, Python packages)
– Developer chooses a project folder → installs its `requirements.txt` or runs `setup.py` → runs experiments via `run.sh` or notebook
– CI/CD (triggered by git push) → checks out monorepo → for each project with `requirements.txt` or `setup.py` → installs dependencies → runs unit tests and smoke scripts → reports status back to GitHub
– Some projects call out to external dataset download scripts or TFHub checkpoints

c. Architectural patterns & design principles
– Monorepo modularity: each project is self‐contained, versioned together for reproducibility
– Convention-over-configuration: consistent folder layout (code, data, notebooks, tests, scripts)
– Clear separation of concerns: root handles global CI/docs, subfolders handle local dependencies
– Leverage standard tooling: Python packaging, Bazel where needed, Travis for CI

d. Key technologies & frameworks
– Python 3.x, Jupyter Notebooks
– ML frameworks: JAX, PyTorch, TensorFlow
– Packaging: `requirements.txt`, `setup.py`, Bazel (`BUILD.bazel`), `pip`
– CI/CD: TravisCI (`.travis.yml`)
– Shell scripts (`run.sh`, `download_*.sh`) for data & experiment orchestration

##### Elements
– Clear labels: “Monorepo Root”, “Project X Package”, “CI/CD”, “Developer Env”, “External Dataset Host”
- Developer → Clone & Env Setup → Run Project Scripts
- GitHub Push → CI/CD → Install → Test → Report
- Project Scripts → External Data Download → Local Data Folder
- Rectangles for code packages (one per project)
- Rounded rectangle for CI/CD pipeline

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Flow
    Dev["Developer Environment"]:::dev
    Dev -->|"git clone"| Clone["Clone Monorepo"]:::action
    Clone --> EnvSetup["Setup Virtual Env & Install Tools"]:::action
    EnvSetup --> SelectProj["Select Project Folder"]:::action
    SelectProj --> AR_Run
    SelectProj --> AVAE_Run
    SelectProj --> DFA_Run
    SelectProj --> WG_Main

    %% Monorepo Root
    subgraph "Monorepo Root"
        MR["Monorepo Root"]:::codePkg
        README["README.md"]:::doc
        TRAVIS[".travis.yml"]:::pipeline
        CONTRIB["CONTRIBUTING.md"]:::doc
        LIC["LICENSE"]:::doc
    end
    click README "https://github.com/google-deepmind/deepmind-research/blob/master/README.md"
    click TRAVIS "https://github.com/google-deepmind/deepmind-research/blob/master/.travis.yml"
    click CONTRIB "https://github.com/google-deepmind/deepmind-research/blob/master/CONTRIBUTING.md"
    click LIC "https://github.com/google-deepmind/deepmind-research/tree/master/LICENSE"

    %% CI/CD Pipeline
    GH["GitHub"]:::external
    GH -->|"push"| CICD["CI/CD Pipeline (TravisCI)"]:::pipeline
    click CICD "https://github.com/google-deepmind/deepmind-research/blob/master/.travis.yml"
    CICD -->|"checkout monorepo"| MR
    CICD -->|"install deps & run tests"| AR_Test
    CICD -->|"install deps & run tests"| AVAE_Test
    CICD -->|"install deps & run tests"| DFA_Test
    CICD -->|"install deps & run tests"| WG_Test
    CICD -->|"report status"| GH

    %% Adversarial Robustness Project
    subgraph "Adversarial Robustness" 
        AR_Code["Source (jax/*.py)"]:::codePkg
        AR_Test["Tests (jax/experiment_test.py)"]:::codePkg
        AR_PT["PyTorch variant"]:::codePkg
        AR_Req["requirements.txt"]:::codePkg
        AR_Run["run.sh"]:::action
    end
    click AR_Code "https://github.com/google-deepmind/deepmind-research/blob/master/adversarial_robustness/jax/*.py"
    click AR_Test "https://github.com/google-deepmind/deepmind-research/blob/master/adversarial_robustness/jax/experiment_test.py"
    click AR_PT "https://github.com/google-deepmind/deepmind-research/tree/master/adversarial_robustness/pytorch/"
    click AR_Req "https://github.com/google-deepmind/deepmind-research/blob/master/adversarial_robustness/requirements.txt"
    click AR_Run "https://github.com/google-deepmind/deepmind-research/blob/master/adversarial_robustness/run.sh"

    %% AVAE Project
    subgraph "AVAE"
        AVAE_Code["vae.py, train.py, train_main.py"]:::codePkg
        AVAE_Util["checkpointer, data_iterators, types"]:::codePkg
        AVAE_Req["requirements.txt"]:::codePkg
        AVAE_Run["run.sh"]:::action
        AVAE_Test["tests"]:::codePkg
    end
    click AVAE_Code "https://github.com/google-deepmind/deepmind-research/blob/master/avae/vae.py"
    click AVAE_Code "https://github.com/google-deepmind/deepmind-research/blob/master/avae/train.py"
    click AVAE_Code "https://github.com/google-deepmind/deepmind-research/blob/master/avae/train_main.py"
    click AVAE_Util "https://github.com/google-deepmind/deepmind-research/blob/master/avae/checkpointer.py"
    click AVAE_Util "https://github.com/google-deepmind/deepmind-research/blob/master/avae/data_iterators.py"
    click AVAE_Util "https://github.com/google-deepmind/deepmind-research/blob/master/avae/types.py"
    click AVAE_Req "https://github.com/google-deepmind/deepmind-research/blob/master/avae/requirements.txt"
    click AVAE_Run "https://github.com/google-deepmind/deepmind-research/blob/master/avae/run.sh"

    %% Density Functional Approximation DM21
    subgraph "Density Functional Approximation DM21"
        DFA_Bazel["BUILD.bazel, WORKSPACE.bazel, .bazelrc"]:::codePkg
        DFA_CPP["C++ examples (cc/*.cc, *.h)"]:::codePkg
        DFA_Py["Python package (*.py)"]:::codePkg
        DFA_CKPT["Checkpoints (data assets)"]:::data
        DFA_Req["requirements.txt, requirements_aot_compilation.txt"]:::codePkg
        DFA_Run["run.sh"]:::action
        DFA_Test["tests"]:::codePkg
    end
    click DFA_Bazel "https://github.com/google-deepmind/deepmind-research/blob/master/density_functional_approximation_dm21/BUILD.bazel"
    click DFA_Bazel "https://github.com/google-deepmind/deepmind-research/blob/master/density_functional_approximation_dm21/.bazelrc"
    click DFA_Bazel "https://github.com/google-deepmind/deepmind-research/blob/master/density_functional_approximation_dm21/WORKSPACE.bazel"
    click DFA_CPP "https://github.com/google-deepmind/deepmind-research/tree/master/density_functional_approximation_dm21/cc/"
    click DFA_Py "https://github.com/google-deepmind/deepmind-research/blob/master/density_functional_approximation_dm21/density_functional_approximation_dm21/*.py"
    click DFA_CKPT "https://github.com/google-deepmind/deepmind-research/tree/master/density_functional_approximation_dm21/density_functional_approximation_dm21/checkpoints/"
    click DFA_Req "https://github.com/google-deepmind/deepmind-research/blob/master/density_functional_approximation_dm21/requirements.txt"
    click DFA_Run "https://github.com/google-deepmind/deepmind-research/blob/master/density_functional_approximation_dm21/run.sh"

    %% WikiGraphs Project
    subgraph "WikiGraphs"
        WG_Main["main.py, setup.py"]:::codePkg
        WG_Scripts["scripts/*.py"]:::codePkg
        WG_Pkg["wikigraphs package"]:::codePkg
        WG_Data["data layer"]:::codePkg
        WG_Model["model layer"]:::codePkg
        WG_Req["requirements.txt"]:::codePkg
        WG_Test["tests"]:::codePkg
    end
    click WG_Main "https://github.com/google-deepmind/deepmind-research/blob/master/wikigraphs/main.py"
    click WG_Main "https://github.com/google-deepmind/deepmind-research/blob/master/wikigraphs/setup.py"
    click WG_Scripts "https://github.com/google-deepmind/deepmind-research/blob/master/wikigraphs/scripts/*.py"
    click WG_Pkg "https://github.com/google-deepmind/deepmind-research/tree/master/wikigraphs/wikigraphs/"
    click WG_Data "https://github.com/google-deepmind/deepmind-research/blob/master/wikigraphs/wikigraphs/data/*.py"
    click WG_Model "https://github.com/google-deepmind/deepmind-research/blob/master/wikigraphs/wikigraphs/model/*.py"
    click WG_Req "https://github.com/google-deepmind/deepmind-research/blob/master/wikigraphs/requirements.txt"

    %% External Services
    subgraph "External Hosts/Services"
        GH2["GitHub"]:::external
        TFH["TFHub"]:::external
        ExtData["External Dataset Hosts"]:::external
    end
    AR_Run -->|"download data"| ExtData
    AVAE_Run -->|"download data"| ExtData
    DFA_Run -->|"download checkpoints"| TFH
    WG_Scripts -->|"fetch data"| ExtData

    %% Styles
    classDef codePkg fill:#D6EAF8,stroke:#2874A6,color:#1B4F72;
    classDef data fill:#FCF3CF,stroke:#B7950B,color:#7D6608,stroke-dasharray: 5 5;
    classDef external fill:#EBDEF0,stroke:#8E44AD,color:#512E5F;
    classDef pipeline fill:#D5F5E3,stroke:#27AE60,color:#1D8348,stroke-width:2px;
    classDef dev fill:#FADBD8,stroke:#C0392B,color:#78281F;
    classDef action fill:#FDEBD0,stroke:#D35400,color:#7E5109;
```
