# ufoym/deepo architecture

> Architecture diagram of the GitHub repository ufoym/deepo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ufoym/deepo
- Repository: https://github.com/ufoym/deepo

## Overview

##### Project Type and Purpose
- Deepo is an open-source CLI framework (generator) for assembling custom Docker images tailored to deep learning environments.
- It provides both a “lego-module” generator to produce Dockerfiles and a set of pre-built Docker images published on Docker Hub.
- Key use cases:
- Quickly pull or run all-in-one or framework-specific Docker images for research.
- Customize and build your own images by selecting only the frameworks you need.

##### High-Level File Structure
- Top-level CI/CD and config
- .github/workflows: CI pipelines (build, test, release)
- .gitignore, LICENSE, README.md, _config.yml
- Docker image definitions
- docker/*.Dockerfile.*: pre-built image templates for CPU/GPU, various frameworks
- Generator code (core + modules)
- generator/core/composer.py: core logic, dependency resolution, topological sort
- generator/generate.py: CLI entrypoint
- generator/modules/*.py: framework-specific module definitions (TensorFlow, PyTorch, etc.)
- Helper scripts
- scripts/*.sh, scripts/*.py: build orchestration, cleaning, CI helper, Docker image tagging

##### Architectural Overview
A. Repository & CI/CD Layer
- GitHub repo (source code)
- GitHub Actions workflows (.github/workflows)
- CI jobs: lint, unit tests of generator, automated Docker builds, Docker Hub pushes
B. Generator Service (CLI)
- CLI entrypoint (generate.py)
- Core component (composer.py): resolves module dependencies via topological sort
- Module registry (generator/modules): plugin modules for each DL framework, Jupyter, tools
C. Dockerfile Generation & Build
- Generated Dockerfile artifact
- Build scripts (scripts/build.sh, scripts/gen-docker.sh) that wrap `docker build`
D. Docker Registry & Runtime
- Docker Hub (ufoym/deepo) for pre-built images
- Users pulling images via `docker pull` or running containers (with GPU via nvidia-docker)
E. Host Environment
- Developer machine / CI runner with Docker engine & optional NVIDIA runtime
- Mounted volumes, port mapping for Jupyter

##### Relationships and Data Flow
1. Developer pushes code → GitHub Actions triggers CI pipelines
2. CI invokes unit tests on generator, then uses scripts/gen-docker.sh to build base images per Dockerfile templates under `docker/`
3. On successful build → CI pushes images to Docker Hub.
4. End user invokes `python generate.py Dockerfile <module-list> [--cuda-ver/CUDNN/Python]` on host →
- CLI loads modules from `generator/modules`, passes them to composer
- Composer topologically sorts modules, resolves inter-dependencies
- Generates a Dockerfile, outputs to local directory
5. User runs `docker build -t my/deepo .` via build.sh → Docker engine creates image, cached layers from base images
6. User runs container: `docker run [--gpus all] -it ufoym/deepo bash` or Jupyter commands

##### Architectural Patterns & Design Principles
- Plugin module pattern: each framework is encapsulated in its own module file under `generator/modules`
- Dependency graph & topological sorting for automatic resolution of install order
- Separation of concerns: generator logic vs. module definitions vs. CI/build scripts
- Infrastructure as code: Dockerfiles for reproducible environments, automated via scripts and CI/CD

##### Technologies & Frameworks
- Python 3.8: generator and scripts
- Docker & NVIDIA Docker: container builds and GPU support
- GitHub Actions: CI/CD automation
- Topological sorting algorithm (in composer.py)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Source & CI/CD Layer
    subgraph "Source & CI/CD"
        direction TB
        GH["GitHub Repo"]:::external
        WF1["dockerimage.yml"]:::ci
        WF2["issue.yml"]:::ci
    end

    GH -->|"git push"| WF1
    GH -->|"git push"| WF2

    %% Generator (CLI)
    subgraph "Generator (CLI)"
        direction TB
        CLI["generator/generate.py"]:::code
        Composer["generator/core/composer.py"]:::code
        Modules["generator/modules/"]:::code
    end

    GH -->|"CI triggers"| CLI
    CLI -->|"loads modules"| Modules
    CLI -->|"invokes sorting"| Composer
    Composer -->|"generates"| GenDockerfile
    Modules -->|"definitions"| Composer

    %% Generated Dockerfile
    GenDockerfile["Generated Dockerfile"]:::build

    CLI -->|"outputs"| GenDockerfile

    %% Build Infrastructure
    subgraph "Build Infrastructure"
        direction TB
        DockerTemplates["docker/"]:::code
        ScriptsBuild["scripts/build.sh\nscripts/gen-docker.sh"]:::code
        ScriptsMakeGD["scripts/make-gen-docker.py"]:::code
        ScriptsMakeCI["scripts/make-ci.py"]:::code
        ScriptsClean["scripts/clean.sh"]:::code
    end

    WF1 -->|"runs make-gen-docker"| ScriptsMakeGD
    WF2 -->|"runs make-ci"| ScriptsMakeCI
    ScriptsMakeGD -->|"process templates"| DockerTemplates
    DockerTemplates -->|"Dockerfile.*"| ScriptsBuild
    GenDockerfile -->|"uses build scripts"| ScriptsBuild
    ScriptsClean -->|"cleanup"| DockerTemplates

    ScriptsBuild -->|"docker build"| LocalImage

    %% Registry & Runtime
    subgraph "Registry & Runtime"
        direction TB
        LocalImage["Temporary Image"]:::build
        DockerHub["Docker Hub"]:::external
    end

    LocalImage -->|"docker push"| DockerHub

    %% Host Environment
    subgraph "Host Environment"
        direction TB
        Host["Developer / CI Runner"]:::external
        DockerEngine["Docker Engine"]:::build
        NVIDIA["NVIDIA Runtime"]:::build
        Volumes["Volumes & Ports"]:::build
    end

    Host -->|"python generate.py"| CLI
    Host -->|"./scripts/build.sh"| ScriptsBuild
    Host --> DockerEngine
    Host --> NVIDIA
    Host --> Volumes
    Host -->|"docker pull/run"| DockerHub
    Host -->|"docker run"| LocalImage

    %% Click Events
    click WF1 "https://github.com/ufoym/deepo/blob/master/.github/workflows/dockerimage.yml"
    click WF2 "https://github.com/ufoym/deepo/blob/master/.github/workflows/issue.yml"
    click CLI "https://github.com/ufoym/deepo/blob/master/generator/generate.py"
    click Composer "https://github.com/ufoym/deepo/blob/master/generator/core/composer.py"
    click Modules "https://github.com/ufoym/deepo/tree/master/generator/modules/"
    click DockerTemplates "https://github.com/ufoym/deepo/tree/master/docker/"
    click ScriptsBuild "https://github.com/ufoym/deepo/blob/master/scripts/build.sh"
    click ScriptsBuild "https://github.com/ufoym/deepo/blob/master/scripts/gen-docker.sh"
    click ScriptsMakeGD "https://github.com/ufoym/deepo/blob/master/scripts/make-gen-docker.py"
    click ScriptsMakeCI "https://github.com/ufoym/deepo/blob/master/scripts/make-ci.py"
    click ScriptsClean "https://github.com/ufoym/deepo/blob/master/scripts/clean.sh"

    %% Styles
    classDef code fill:#cfe2f3,stroke:#333,stroke-width:1px
    classDef build fill:#d9ead3,stroke:#333,stroke-width:1px
    classDef ci fill:#f9cb9c,stroke:#333,stroke-width:1px
    classDef external fill:#d9d9d9,stroke:#333,stroke-width:1px
```
