# demisto/dockerfiles architecture

> Architecture diagram of the GitHub repository demisto/dockerfiles, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/demisto/dockerfiles
- Repository: https://github.com/demisto/dockerfiles

## Overview

- Hundreds of directories inside a top-level docker folder—each representing a different Docker image (with its own Dockerfile, Pipfile/pyproject.toml, build.conf, and sometimes verify scripts).
- Numerous automation scripts (such as create_new_docker_image.py, build_docker.sh, and update scripts) and utilities to generate configuration files, update dependencies (for instance Dependabot, native Docker validations, versioning scripts, etc.).
- CI/CD configurations spread across .github (including multiple workflows) and .gitlab directories that integrate with GitHub Actions and GitLab CI.
- Testing and validation components (unit_tests folder, verify.py, verify.ps1) that ensure the built images meet requirements.
- Helper utilities in the utils folder to manage Dockerfile updates and dependency comparisons.

##### Main Components
a. Repository & Source Control:
– A central repository (monorepo) that holds all Docker image definitions and helper tools.
– Integration with CI/CD (GitHub workflows, GitLab CI) and automated dependency management via Dependabot.
b. Docker Image Modules:
– Each folder under “docker” represents one image/micro-module.
– Inside each module, files such as Dockerfile, build.conf, dynamic_version.sh, and verify scripts define the image’s build, versioning, and validation logic.
c. Build & Deployment Engine:
– The build_docker.sh script which scans for changes (using git), triggers builds, and pushes images to the appropriate Docker Hub organizations (devdemisto for PR testing; demisto for production).
– Helper scripts (create_new_docker_image.py, update_dependabot.sh) that automate creating new image directories or updating configurations.
d. Dependency & Environment Management:
– Usage of pipenv and poetry (with Pipfile, Pipfile.lock, or pyproject.toml) to lock and export dependencies.
– Generation of requirements.txt files that are subsequently used in Docker build instructions.
e. CI/CD Workflows:
– GitHub Actions workflows initiate builds for modified images.
– GitLab CI scripts provide additional integration tests and helper functions.
f. Testing & Verification:
– Verify scripts (verify.py or verify.ps1) in Docker images for post-build validation.
– Unit tests (under unit_tests) and integration tests that ensure version and license checks.
g. Utilities and Automation:
– Tools within the utils folder that help to update Dockerfiles, compare dependency constraints, and manage deprecated image records.

##### Relationships & Data/Control Flow
– A developer commits a change (for example, to Dockerfiles or configuration files) in the central repository.
– CI/CD workflows (triggered via GitHub Actions or GitLab CI) notice the change and run build scripts.
– The build_docker.sh orchestrates which image folders are “dirty” by performing a git diff against origin/master.
– For each changed image module, the system:
- Uses the Dockerfile and configuration (build.conf, dynamic_version.sh) to generate the build context.
- Leverages pipenv/poetry to generate a deterministic list of dependencies.
- Runs the Docker build and then, if present, executes the verify scripts to validate the container.
– After a successful build, the images are tagged (with version and commit information) and then deployed to the appropriate Docker Hub organization.
– Additionally, automated scripts (with Dependabot integrations and common dependency installers) keep dependencies up to date and licenses approved.

##### Architectural Patterns & Principles
– Monorepo: All components for the Docker image building process coexist in one repository to foster consistent configurations and shared tooling.
– Modularization: Each Docker image is encapsulated in its own folder, making it an independently buildable module with its configuration.
– CI/CD Automation: Heavy reliance on automated workflows using GitHub Actions and GitLab CI to ensure cohesive mirror deployments between development and production images.
– Dependency Management: Using pipenv/poetry and automated export processes to guarantee stable, reproducible builds.
– Validation & Testing: Integration of verify scripts and unit tests to immediately detect any misconfiguration or build failures.
– Extensibility: Scripts for creating new images and updating configurations allow the system to grow and incorporate many new Docker modules for various integration needs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source & CI/CD Subgraph
    subgraph "Source & CI/CD"
        DEV["Developer"]:::external
        REP["Monorepo: Repository & Source Control"]:::repo
        CICD["CI/CD Workflows (GitHub Actions,GitLab CI)"]:::ci
        DEV -->|"commits"| REP
        REP -->|"triggers"| CICD
    end

    %% Build & Deployment Engine Subgraph
    subgraph "Build & Deployment Engine"
        BUILD1["build_docker.sh"]:::build
        BUILD2["create_new_docker_image.py"]:::build
        CICD -->|"initiates"| BUILD1
        BUILD1 -->|"calls"| BUILD2
    end

    %% Build Components Subgraph
    subgraph "Build Components"
        MODULES["Docker Image Modules"]:::module
        DEP["Dependency Management (Pipenv/Poetry Files)"]:::dep
        TEST["Testing & Verification (verify scripts,unit_tests)"]:::test
        DOCKERHUB["Docker Hub Deployment (devdemisto,demisto)"]:::dockerhub
        BUILD2 -->|"uses"| MODULES
        BUILD2 -->|"uses"| DEP
        BUILD2 -->|"validates"| TEST
        TEST -->|"pushes"| DOCKERHUB
    end

    %% Utilities & Automation
    UTIL["Utilities & Automation (update scripts,utils)"]:::util
    BUILD2 -->|"updates"| UTIL

    %% Click Events for Component Mapping
    click MODULES "https://github.com/demisto/dockerfiles/tree/master/docker/"
    click BUILD1 "https://github.com/demisto/dockerfiles/blob/master/docker/build_docker.sh"
    click BUILD2 "https://github.com/demisto/dockerfiles/blob/master/docker/create_new_docker_image.py"
    click CICD "https://github.com/demisto/dockerfiles/tree/master/.github/workflows"
    click CICD "https://github.com/demisto/dockerfiles/blob/master/.gitlab"
    click DEP "https://github.com/demisto/dockerfiles/tree/master/Pipfile"
    click DEP "https://github.com/demisto/dockerfiles/blob/master/Pipfile.lock"
    click DEP "https://github.com/demisto/dockerfiles/blob/master/pyproject.toml"
    click DEP "https://github.com/demisto/dockerfiles/blob/master/poetry.toml"
    click TEST "https://github.com/demisto/dockerfiles/tree/master/unit_tests"
    click TEST "https://github.com/demisto/dockerfiles/blob/master/docker/verify.py"
    click TEST "https://github.com/demisto/dockerfiles/blob/master/docker/powershell-core/verify.ps1"
    click UTIL "https://github.com/demisto/dockerfiles/tree/master/utils/"
    click UTIL "https://github.com/demisto/dockerfiles/blob/master/docker/verify_dependabot.sh"
    click UTIL "https://github.com/demisto/dockerfiles/blob/master/docker/update_images_from_crypto.sh"

    %% Styles & Classes
    classDef external fill:#f9c74f,stroke:#333,stroke-width:2px
    classDef repo fill:#90be6d,stroke:#333,stroke-width:2px
    classDef ci fill:#577590,stroke:#fff,stroke-width:2px,color:#fff
    classDef build fill:#f94144,stroke:#333,stroke-width:2px,color:#fff
    classDef module fill:#277da1,stroke:#fff,stroke-width:2px,color:#fff
    classDef dep fill:#f3722c,stroke:#fff,stroke-width:2px,color:#fff
    classDef test fill:#90be6d,stroke:#333,stroke-width:2px,color:#fff
    classDef util fill:#43aa8b,stroke:#333,stroke-width:2px,color:#fff
    classDef dockerhub fill:#f9c74f,stroke:#333,stroke-width:2px
```
