How ufoym/deepo works
Project Type and Purpose
• Deepo is an open-source CLI framework (generator) for assembling custom Docker images tailored to deep learning environments.
• It provides both a “lego-module” generator to produce Dockerfiles and a set of pre-built Docker images published on Docker Hub.
• Key use cases:
• Quickly pull or run all-in-one or framework-specific Docker images for research.
• Customize and build your own images by selecting only the frameworks you need.
High-Level File Structure
• Top-level CI/CD and config
• .github/workflows: CI pipelines (build, test, release)
• .gitignore, LICENSE, README.md, _config.yml
• Docker image definitions
• docker/*.Dockerfile.*: pre-built image templates for CPU/GPU, various frameworks
• Generator code (core + modules)
• generator/core/composer.py: core logic, dependency resolution, topological sort
• generator/generate.py: CLI entrypoint
• generator/modules/*.py: framework-specific module definitions (TensorFlow, PyTorch, etc.)
• Helper scripts
• scripts/*.sh, scripts/*.py: build orchestration, cleaning, CI helper, Docker image tagging
Architectural Overview
A. Repository & CI/CD Layer
• GitHub repo (source code)
• GitHub Actions workflows (.github/workflows)
• CI jobs: lint, unit tests of generator, automated Docker builds, Docker Hub pushes
B. Generator Service (CLI)
• CLI entrypoint (generate.py)
• Core component (composer.py): resolves module dependencies via topological sort
• Module registry (generator/modules): plugin modules for each DL framework, Jupyter, tools
C. Dockerfile Generation & Build
• Generated Dockerfile artifact
• Build scripts (scripts/build.sh, scripts/gen-docker.sh) that wrap docker build
D. Docker Registry & Runtime
• Docker Hub (ufoym/deepo) for pre-built images
• Users pulling images via docker pull or running containers (with GPU via nvidia-docker)
E. Host Environment
• Developer machine / CI runner with Docker engine & optional NVIDIA runtime
• Mounted volumes, port mapping for Jupyter
Relationships and Data Flow
1. Developer pushes code → GitHub Actions triggers CI pipelines
2. CI invokes unit tests on generator, then uses scripts/gen-docker.sh to build base images per Dockerfile templates under docker/
3. On successful build → CI pushes images to Docker Hub.
4. End user invokes python generate.py Dockerfile <module-list> [--cuda-ver/CUDNN/Python] on host →
• CLI loads modules from generator/modules, passes them to composer
• Composer topologically sorts modules, resolves inter-dependencies
• Generates a Dockerfile, outputs to local directory
5. User runs docker build -t my/deepo . via build.sh → Docker engine creates image, cached layers from base images
6. User runs container: docker run [--gpus all] -it ufoym/deepo bash or Jupyter commands
Architectural Patterns & Design Principles
• Plugin module pattern: each framework is encapsulated in its own module file under generator/modules
• Dependency graph & topological sorting for automatic resolution of install order
• Separation of concerns: generator logic vs. module definitions vs. CI/build scripts
• Infrastructure as code: Dockerfiles for reproducible environments, automated via scripts and CI/CD
Technologies & Frameworks
• Python 3.8: generator and scripts
• Docker & NVIDIA Docker: container builds and GPU support
• GitHub Actions: CI/CD automation
• Topological sorting algorithm (in composer.py)