How esl-epfl/szcore works
Main Components
• Repository & Version Control – Source code, configuration files, algorithm YAML submissions, and website files are all managed in Git.
• Algorithm Submission Interface – Developers add a new YAML file (and submit their containerized algorithm image) into the “algorithms” (and optionally “challenge_algorithms”) directory.
• CI/CD Pipeline – GitHub Actions workflows (listed under .github/workflows) are used to validate PRs (“1-pr-check.yml”), build and deploy the website (“2-build-website.yml”), and to handle Docker image verification (docker-publish-algo-checks.yml).
• Algorithm Container Execution Environment – Docker images built using the provided Dockerfile templates (in config) execute the submitted algorithms offline. They run inference, use predefined paths for I/O, and compute performance metrics.
• Benchmarking Engine – Once merged, the CI jobs pull the Docker images, execute algorithms on benchmark datasets, and produce performance metrics (whose outputs are stored in the “results” directory as JSON files).
• Static Website – The “website” directory contains the static site content (assets, Hugo configuration, layouts, themes) that displays benchmark results, algorithm details, datasets, and challenge descriptions.
Relationships and Interactions
• The user (algorithm developer) submits a YAML file and links to their Docker image via a PR.
• The submission triggers the CI/CD workflows; one branch verifies the submission while another builds and deploys the website.
• The Docker image (built according to the provided template) is executed by the CI pipeline, which runs benchmarks using input data stored in “tests” and references datasets defined in “website/content/datasets”.
• Benchmark results are generated and stored in the “results” folder and then pushed/updated on the static website.
• The website serves as an interface for viewing the detailed performance statistics of submitted algorithms.
Architectural Patterns and Design Principles
• Modular design with clear separation of concerns: The repository is split between algorithm definitions, site content, CI/CD configurations, and benchmarking data.
• Containerization: The use of Docker ensures that algorithm execution is isolated and standardized.
• Automation through CI/CD: GitHub Actions manages continuous integration, validation, and deployment cycles.
• Static Website Generation: Use of Hugo and static asset management for front-end presentation simplifies the deployment process.
• Open source extensibility: New algorithms and challenges are introduced by adding new YAML files (using a consistent schema) and corresponding Docker images.
Technologies, Frameworks, and Libraries
• GitHub Actions as the orchestration/automation tool.
• Docker for containerization (with Dockerfiles provided in the “config” directory).
• YAML as the submission configuration format.
• Hugo and Tailwind CSS (inside the “website” directory) for building the static website.
• Python scripts (in “website/python_scripts”) for parsing and processing benchmark results.
• Various JSON files in “results” and “website/static” to store and serve performance metrics.
• Submission (developers pushing PRs) → CI/CD jobs →
• Containerized Testing/Benchmarking (using Docker images based on the template) →
• Results Output (JSON files saved in the “results” directory) →
• Website Update (static site regeneration and deployment).
• External actors like “Algorithm Developer” (initiates a PR) and “User/Researcher” (viewing the website).