How demisto/dockerfiles works
• Hundreds of directories inside a top-level docker folder—each representing a different Docker image (with its own Dockerfile, Pipfile/pyproject.toml, build.conf, and sometimes verify scripts).
• Numerous automation scripts (such as create_new_docker_image.py, build_docker.sh, and update scripts) and utilities to generate configuration files, update dependencies (for instance Dependabot, native Docker validations, versioning scripts, etc.).
• CI/CD configurations spread across .github (including multiple workflows) and .gitlab directories that integrate with GitHub Actions and GitLab CI.
• Testing and validation components (unit_tests folder, verify.py, verify.ps1) that ensure the built images meet requirements.
• Helper utilities in the utils folder to manage Dockerfile updates and dependency comparisons.
Main Components
a. Repository & Source Control:
– A central repository (monorepo) that holds all Docker image definitions and helper tools.
– Integration with CI/CD (GitHub workflows, GitLab CI) and automated dependency management via Dependabot.
b. Docker Image Modules:
– Each folder under “docker” represents one image/micro-module.
– Inside each module, files such as Dockerfile, build.conf, dynamic_version.sh, and verify scripts define the image’s build, versioning, and validation logic.
c. Build & Deployment Engine:
– The build_docker.sh script which scans for changes (using git), triggers builds, and pushes images to the appropriate Docker Hub organizations (devdemisto for PR testing; demisto for production).
– Helper scripts (create_new_docker_image.py, update_dependabot.sh) that automate creating new image directories or updating configurations.
d. Dependency & Environment Management:
– Usage of pipenv and poetry (with Pipfile, Pipfile.lock, or pyproject.toml) to lock and export dependencies.
– Generation of requirements.txt files that are subsequently used in Docker build instructions.
e. CI/CD Workflows:
– GitHub Actions workflows initiate builds for modified images.
– GitLab CI scripts provide additional integration tests and helper functions.
f. Testing & Verification:
– Verify scripts (verify.py or verify.ps1) in Docker images for post-build validation.
– Unit tests (under unit_tests) and integration tests that ensure version and license checks.
g. Utilities and Automation:
– Tools within the utils folder that help to update Dockerfiles, compare dependency constraints, and manage deprecated image records.
Relationships & Data/Control Flow
– A developer commits a change (for example, to Dockerfiles or configuration files) in the central repository.
– CI/CD workflows (triggered via GitHub Actions or GitLab CI) notice the change and run build scripts.
– The build_docker.sh orchestrates which image folders are “dirty” by performing a git diff against origin/master.
– For each changed image module, the system:
• Uses the Dockerfile and configuration (build.conf, dynamic_version.sh) to generate the build context.
• Leverages pipenv/poetry to generate a deterministic list of dependencies.
• Runs the Docker build and then, if present, executes the verify scripts to validate the container.
– After a successful build, the images are tagged (with version and commit information) and then deployed to the appropriate Docker Hub organization.
– Additionally, automated scripts (with Dependabot integrations and common dependency installers) keep dependencies up to date and licenses approved.
Architectural Patterns & Principles
– Monorepo: All components for the Docker image building process coexist in one repository to foster consistent configurations and shared tooling.
– Modularization: Each Docker image is encapsulated in its own folder, making it an independently buildable module with its configuration.
– CI/CD Automation: Heavy reliance on automated workflows using GitHub Actions and GitLab CI to ensure cohesive mirror deployments between development and production images.
– Dependency Management: Using pipenv/poetry and automated export processes to guarantee stable, reproducible builds.
– Validation & Testing: Integration of verify scripts and unit tests to immediately detect any misconfiguration or build failures.
– Extensibility: Scripts for creating new images and updating configurations allow the system to grow and incorporate many new Docker modules for various integration needs.