How google-deepmind/deepmind-research works
Project type and purpose
• This is a monolithic “monorepo” hosting dozens of independent research projects and accompanying code, notebooks, data, and scripts.
• Each top‐level folder (e.g., adversarial_robustness, avae, fusion_tcv, wikigraphs, …) is a self‐contained research package illustrating DeepMind work in a particular paper.
• The root README.md, .travis.yml, LICENSE, CONTRIBUTING.md describe the umbrella repository and its CI/CD.
File structure
• Top level:
• Monorepo root (contains global CI config and docs)
• One folder per project (each a Python package or demo)
• Within each project folder you typically find:
• Source code modules (*.py, sometimes subpackages)
• Jupyter notebooks (*.ipynb) for demos/analysis
• Data or checkpoint assets (e.g. *.npz, TFRecords)
• Entry scripts (run.sh, main.py, experiment drivers)
• Packaging support (requirements.txt, setup.py, BUILD.bazel)
• Unit tests (*_test.py)
• There is no cross‐project shared library – each project depends only on its own declared requirements.
README for additional insights
• The root README lists ~60 projects, each tied to a publication.
• Many use common ML frameworks (JAX, PyTorch, TensorFlow) but keep their codebases isolated.
• Some use Bazel (e.g. density_functional_approximation_dm21), most use standard Python packaging.
a. Main components
1. Monorepo Root (“DeepMind Research”)
2. CI/CD Pipeline (TravisCI)
3. Developer Environment (local or cloud dev machine, conda/venv)
4. Individual Project Packages (one per paper)
– Source code
– Notebooks
– Data assets
– Scripts & tests
5. Dependency Manager & Build Tools (pip, Bazel)
6. External Hosts/Services (GitHub, TFHub, external datasets)
b. Relationships & interactions
– Developer clones monorepo → creates virtual environment → installs global tools (e.g. Bazel, Python packages)
– Developer chooses a project folder → installs its requirements.txt or runs setup.py → runs experiments via run.sh or notebook
– CI/CD (triggered by git push) → checks out monorepo → for each project with requirements.txt or setup.py → installs dependencies → runs unit tests and smoke scripts → reports status back to GitHub
– Some projects call out to external dataset download scripts or TFHub checkpoints
c. Architectural patterns & design principles
– Monorepo modularity: each project is self‐contained, versioned together for reproducibility
– Convention-over-configuration: consistent folder layout (code, data, notebooks, tests, scripts)
– Clear separation of concerns: root handles global CI/docs, subfolders handle local dependencies
– Leverage standard tooling: Python packaging, Bazel where needed, Travis for CI
d. Key technologies & frameworks
– Python 3.x, Jupyter Notebooks
– ML frameworks: JAX, PyTorch, TensorFlow
– Packaging: requirements.txt, setup.py, Bazel (BUILD.bazel), pip
– CI/CD: TravisCI (.travis.yml)
– Shell scripts (run.sh, download_*.sh) for data & experiment orchestration
Elements
– Clear labels: “Monorepo Root”, “Project X Package”, “CI/CD”, “Developer Env”, “External Dataset Host”
• Developer → Clone & Env Setup → Run Project Scripts
• GitHub Push → CI/CD → Install → Test → Report
• Project Scripts → External Data Download → Local Data Folder
• Rectangles for code packages (one per project)
• Rounded rectangle for CI/CD pipeline