# augmentcode/environments architecture

> Architecture diagram of the GitHub repository augmentcode/environments, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/augmentcode/environments
- Repository: https://github.com/augmentcode/environments

## Overview

##### Project type and purpose
- This is an open-source DevOps automation tool for building, packaging, and publishing Docker environment images (CPU/GPU/ROCm) and provisioning related AWS AMIs via Packer+Ansible.
- It integrates with CI/CD (CircleCI) and Docker registries (DockerHub, NGC), and coordinates with the main Determined AI codebase for testing image compatibility.

##### File structure
- Top-level build artefacts:
- Dockerfiles (`Dockerfile-*.cpu`, `Dockerfile-*.gpu`, etc.)
- `Makefile` orchestrating builds/publishes
- `dockerfile_scripts/` containing helper scripts & patches
- `scripts/` for pushing images
- Cloud provisioning:
- `cloud/environments-packer.json` for Packer
- `cloud/environments-playbook.yml` and `cloud/roles/` for Ansible roles (e.g. aws-fs, docker, gpumon, journald-cloudwatch, nvidia-drivers, sysctl)
- CI/CD configs: `.circleci/config.yml`
- Versioning & metadata: `.bumpversion.cfg`, `VERSION`, `version-matrix.yaml`

##### README for additional insights
- The repository’s main workflow:
a. Developer modifies Dockerfile or scripts →
b. `make build` → build local image(s) →
c. `make publish` → push to specified registry →
d. (Optionally) trigger CI jobs for downstream testing →
e. Use Packer+Ansible to bake cloud AMIs / configure AWS instances.
- Multi-platform build approach via Docker Buildx for arm64/x86_64.

##### Main components & interactions
a. Developer Local Machine
- Edits files, runs `make`, invokes Docker, Buildx, Packer, Ansible.
b. Source Control (GitHub)
- Hosts repo, CI triggers on PRs, push events.
c. CI/CD (CircleCI)
- Validates Docker builds, runs tests, approves publishing.
d. Build System
- Makefile orchestrates:
- Docker build contexts (Dockerfiles + dockerfile_scripts)
- Buildx for multi-platform → Docker Images
- scripts/publish-docker.sh → pushes images to registries
- Packer + `cloud/environments-packer.json` → AMI build
- Ansible playbooks (`cloud/environments-playbook.yml`) + `cloud/roles/*` → config AWS instances
e. Artifact Repositories
- DockerHub registry
- NGC registry
- (Optionally) AWS AMI catalog
f. Downstream Integration
- Main Determined AI repo uses these images for ML experiments and e2e tests.

##### Architectural patterns & design principles
- Infrastructure as Code: Ansible roles and Packer templates define cloud infra reproducibly.
- Modular build helpers: `dockerfile_scripts/` encapsulates package installs, patches, making Dockerfiles DRY.
- GitOps/CI-driven deployments: `.circleci` workflows, Makefile targets drive build/test/publish.
- Separation of concerns:
- Build logic (Makefile + Dockerfiles)
- Packaging helpers (`dockerfile_scripts`)
- Publishing scripts (`scripts`)
- Cloud provisioning (`cloud/`)

##### Key elements to capture
- Clear component labels (e.g., “Makefile (build orchestrator)”, “Docker Buildx”)
- Annotations for critical configurations (e.g., `cloud/environments-playbook.yml`, `.circleci/config.yml`)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local Development
    subgraph "Local Development"
        direction TB
        Dev["Developer Local Machine"]:::dev
        GitHub["GitHub (Repo)"]:::dev
    end

    %% CI/CD Pipeline
    subgraph "CI/CD Pipeline"
        direction TB
        CircleCI["CircleCI (.circleci/config.yml)"]:::build
    end

    %% Build Automation
    subgraph "Build Automation"
        direction TB
        Make["Makefile (build orchestrator)"]:::build
        subgraph "Docker Build Contexts & Helpers"
            direction TB
            DF1["Dockerfile-base-cpu"]:::build
            DF2["Dockerfile-base-gpu"]:::build
            DF3["Dockerfile-default-cpu"]:::build
            DF4["Dockerfile-default-gpu"]:::build
            DF5["Dockerfile-default-rocm"]:::build
            DF6["Dockerfile-ngc-hpc"]:::build
            DF7["Dockerfile-pytorch-ngc"]:::build
            DF8["Dockerfile-tensorflow-ngc"]:::build
            Helpers["dockerfile_scripts/"]:::script
        end
        Buildx["Docker Buildx"]:::build
        subgraph "Publishing Scripts"
            direction TB
            Pub1["scripts/publish-docker.sh"]:::script
            Pub2["scripts/publish-versionless-docker.sh"]:::script
        end
    end

    %% Artifact Repositories
    subgraph "Artifact Repositories"
        direction TB
        DH[("DockerHub")]:::repo
        NGC[("NGC Registry")]:::repo
        AMICat[("AWS AMI Catalog")]:::repo
    end

    %% Cloud Provisioning & Monitoring
    subgraph "Cloud Provisioning & Monitoring"
        direction TB
        Packer["Packer (cloud/environments-packer.json)"]:::infra
        Ansible["Ansible Playbook (cloud/environments-playbook.yml)"]:::infra
        Roles["Ansible Roles (cloud/roles/)"]:::infra
        Instance["AWS EC2 Instance"]:::cloud
        Monitors["Monitoring Agents (gpumon, journald-cloudwatch)"]:::cloud
        CW["CloudWatch"]:::cloud
    end

    %% Connections
    Dev -->|"git push / PR"| GitHub
    GitHub -->|"webhook trigger"| CircleCI
    CircleCI -->|"invoke make build"| Make
    Make -->|"uses Dockerfiles & helpers"| DF1 & DF2 & DF3 & DF4 & DF5 & DF6 & DF7 & DF8 & Helpers
    Make -->|"docker buildx"| Buildx
    Buildx -->|"build images"| Pub1 & Pub2
    Pub1 -->|"push images"| DH
    Pub2 -->|"push versionless"| NGC
    Make -->|"packer build"| Packer
    Packer -->|"produce AMI"| AMICat
    Make -->|"ansible-playbook"| Ansible
    Ansible -->|"applies roles"| Roles
    Ansible -->|"provision"| Instance
    Instance -->|"runs"| Monitors
    Monitors -->|"send metrics"| CW

    %% Click Events
    click Make "https://github.com/augmentcode/environments/tree/main/Makefile"
    click DF1 "https://github.com/augmentcode/environments/tree/main/Dockerfile-base-cpu"
    click DF2 "https://github.com/augmentcode/environments/tree/main/Dockerfile-base-gpu"
    click DF3 "https://github.com/augmentcode/environments/tree/main/Dockerfile-default-cpu"
    click DF4 "https://github.com/augmentcode/environments/tree/main/Dockerfile-default-gpu"
    click DF5 "https://github.com/augmentcode/environments/tree/main/Dockerfile-default-rocm"
    click DF6 "https://github.com/augmentcode/environments/tree/main/Dockerfile-ngc-hpc"
    click DF7 "https://github.com/augmentcode/environments/tree/main/Dockerfile-pytorch-ngc"
    click DF8 "https://github.com/augmentcode/environments/tree/main/Dockerfile-tensorflow-ngc"
    click Helpers "https://github.com/augmentcode/environments/tree/main/dockerfile_scripts/"
    click Pub1 "https://github.com/augmentcode/environments/blob/main/scripts/publish-docker.sh"
    click Pub2 "https://github.com/augmentcode/environments/blob/main/scripts/publish-versionless-docker.sh"
    click Packer "https://github.com/augmentcode/environments/blob/main/cloud/environments-packer.json"
    click Ansible "https://github.com/augmentcode/environments/blob/main/cloud/environments-playbook.yml"
    click Roles "https://github.com/augmentcode/environments/tree/main/cloud/roles/"
    click CircleCI "https://github.com/augmentcode/environments/blob/main/.circleci/config.yml"

    %% Styles
    classDef dev fill:#D6E9FF,stroke:#0366D6
    classDef build fill:#DFF0D8,stroke:#3C763D
    classDef script fill:#FCF8E3,stroke:#8A6D3B
    classDef repo fill:#E0F7FA,stroke:#00796B
    classDef infra fill:#FCF8E3,stroke:#8A6D3B
    classDef cloud fill:#FDEBD0,stroke:#E67E22
```
