# google-deepmind/code_contests architecture

> Architecture diagram of the GitHub repository google-deepmind/code_contests, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/code_contests
- Repository: https://github.com/google-deepmind/code_contests

## Overview

##### Project type and purpose
- Type: A data-processing and execution library/toolkit for competitive programming datasets.
- Purpose:
– Provide protocol-buffer definitions and I/O (via Riegeli) for contest problems.
– Offer C++ and Python example clients to iterate over the dataset.
– Supply an execution engine (sandboxed) to compile and run submitted solutions for correctness/evaluation.

##### File structure
- Top-level:
– BUILD files and WORKSPACE: Bazel as build orchestrator
– contest_problem.proto: central data schema
– print_names.cc / print_names_and_sources.py: example ingestion clients
– execution/: C++ sandboxed execution engine (BUILD.bazel, solver, sandboxer, threadpool, tests)
– third_party/: external compression, hashing, and I/O libraries (snappy, zlib, highwayhash, net_zstd, crc32, farmhash)
- Patterns:
– Modular libraries (ingestion vs. execution)
– Clear separation of data schema, I/O, execution, and third-party dependencies.

##### README for technical stack
- Languages: C++17, Python 2.7/3.x
- Build: Bazel on Linux (clang)
- Data format: Riegeli-encoded protocol buffers
- Execution: sandboxing (resource-limited child processes) via tester_sandboxer
- Dependencies: Cloud SDK (gsutil) for data download

a. Data Storage
– GCS bucket “dm-code_contests”
– `.riegeli` files (train/valid/test)
b. Data Schema & I/O Library
– contest_problem.proto → Bazel-built C++/Python bindings
– Riegeli reader/writer (third_party/riegeli)
– Example clients:
- print_names.cc (C++)
- print_names_and_sources.py (Python)
c. Execution Engine
– solve_example binary → CLI front-end
– Execution Controller (solve_example, py_tester_sandboxer)
– Thread Pool (simple_threadpool.h)
– Sandboxer modules (tester_sandboxer.cc/h, py_tester_sandboxer.cc/h)
– Python interpreters (paths from py_locations.cc)
– Status/reporting (status_macros.h, status_matchers.h)
d. Build System & Toolchain
– Bazel WORKSPACE & BUILD.bazel files
– External dependencies under third_party
e. External Tools & Environments
– Cloud SDK (gsutil) for data transfer
– System interpreters: `/usr/bin/python3.9`, `/usr/bin/python2.7`

##### Relationships and data-flow
1. GCS Bucket → Data Downloader (`gsutil cp`) → Local Riegeli files
2. Riegeli files → Example clients (print_names, solve_example)
3. Example clients → I/O Library → contest_problem proto objects
4. solve_example → Execution Engine → Sandboxer → invokes Python processes
5. Sandboxer → returns status/results to solve_example → stdout/log
- Build-time dependencies:
– Bazel orchestrates compilation of ingestion clients, execution engine, and links third-party libs.

##### Architectural patterns & design principles
- Modular layering: Data Schema → I/O → Processing/Execution → Client CLI
- Separation of concerns: ingestion vs. execution vs. tooling
- Pluggable sandbox: both C++ and Python sandbox implementations
- Testability: separate tester_sandboxer_test.cc to validate sandbox behavior

- Clear component labels and boundaries
- Version or platform note (“Linux/clang, Python2.7 & 3.x”)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Storage Layer
    subgraph "Data Storage" 
        direction TB
        GCS["GCS Bucket\n(dm-code_contests)"]:::storage
        RIEGELI[".riegeli Files\n(train/valid/test)"]:::storage
    end

    %% Schema & I/O Layer
    subgraph "Data Schema & I/O Library"
        direction TB
        PROTO["contest_problem.proto\n(schema)"]:::libs
        RIEG_READER["Riegeli Reader/Writer"]:::libs
        THIRD_PARTY["Third-Party Libs\n(snappy,zlib,highwayhash,net_zstd,crc32,farmhash)"]:::libs
    end

    %% Clients Layer
    subgraph "Example Clients"
        direction TB
        CPP_CLIENT["print_names.cc\n(C++ ingestion)"]:::exec
        PY_CLIENT["print_names_and_sources.py\n(Python ingestion)"]:::exec
    end

    %% Execution Engine Layer
    subgraph "Execution Engine"
        direction TB
        SOLVE["solve_example CLI"]:::exec
        CONTROLLER["Execution Controller"]:::exec
        THREADPOOL["simple_threadpool.h"]:::libs
        SANDBOX_CPP["tester_sandboxer.cc/h"]:::libs
        SANDBOX_PY["py_tester_sandboxer.cc/h"]:::libs
        PY_LOC["py_locations.cc/h"]:::libs
        STATUS["status_macros.h\nstatus_matchers.h"]:::libs
        TESTS["tester_sandboxer_test.cc"]:::exec
    end

    %% Build System Layer
    subgraph "Build System & Toolchain"
        direction TB
        WORKSPACE_FILE["WORKSPACE"]:::external
        BUILD_TOP["BUILD"]:::external
        BUILD_EXEC["execution/BUILD.bazel"]:::external
        BUILD_3P["third_party/BUILD.bazel\nsnappy.BUILD.bazel\nzlib.BUILD.bazel\nhighwayhash.BUILD.bazel\nnet_zstd.BUILD.bazel\ncrc32.BUILD.bazel\nfarmhash.BUILD"]:::external
    end

    %% External Tools
    subgraph "External Tools & Environments"
        direction TB
        CLOUD_SDK["Cloud SDK\n(gsutil)"]:::external
        PY_SYS["/usr/bin/python2.7\n/usr/bin/python3.9"]:::external
    end

    %% Runtime Data Flows
    GCS -->|"gsutil cp"| RIEGELI
    RIEGELI -->|"Riegeli read"| CPP_CLIENT
    RIEGELI -->|"Riegeli read"| PY_CLIENT
    CPP_CLIENT -->|uses| PROTO
    PY_CLIENT -->|uses| PROTO
    PROTO -->|generated C++/Python bindings| RIEG_READER
    CPP_CLIENT -->|invokes| SOLVE
    SOLVE -->|controls| CONTROLLER
    CONTROLLER -->|dispatch| SANDBOX_CPP
    CONTROLLER -->|dispatch| SANDBOX_PY
    SANDBOX_PY -->|spawns| PY_SYS
    SANDBOX_CPP -->|uses| THREADPOOL
    SOLVE -->|logs| STATUS
    SOLVE -->|tested by| TESTS

    %% Build-time Dependencies
    WORKSPACE_FILE -.->|orchestrates| BUILD_TOP
    BUILD_TOP -.->|includes| BUILD_EXEC
    BUILD_TOP -.->|includes| BUILD_3P
    BUILD_EXEC -.->|builds| SANDBOX_CPP
    BUILD_EXEC -.->|builds| SANDBOX_PY
    BUILD_EXEC -.->|builds| SOLVE
    BUILD_EXEC -.->|builds| CPP_CLIENT
    BUILD_EXEC -.->|builds| PY_CLIENT

    %% External Tool Interactions
    CLOUD_SDK -->|downloads| RIEGELI

    %% Click Events
    click PROTO "https://github.com/google-deepmind/code_contests/blob/main/contest_problem.proto"
    click CPP_CLIENT "https://github.com/google-deepmind/code_contests/blob/main/print_names.cc"
    click PY_CLIENT "https://github.com/google-deepmind/code_contests/blob/main/print_names_and_sources.py"
    click SOLVE "https://github.com/google-deepmind/code_contests/blob/main/execution/solve_example.cc"
    click SANDBOX_CPP "https://github.com/google-deepmind/code_contests/blob/main/execution/tester_sandboxer.cc"
    click SANDBOX_CPP "https://github.com/google-deepmind/code_contests/blob/main/execution/tester_sandboxer.h"
    click SANDBOX_PY "https://github.com/google-deepmind/code_contests/blob/main/execution/py_tester_sandboxer.cc"
    click SANDBOX_PY "https://github.com/google-deepmind/code_contests/blob/main/execution/py_tester_sandboxer.h"
    click THREADPOOL "https://github.com/google-deepmind/code_contests/blob/main/execution/simple_threadpool.h"
    click PY_LOC "https://github.com/google-deepmind/code_contests/blob/main/execution/py_locations.cc"
    click PY_LOC "https://github.com/google-deepmind/code_contests/blob/main/execution/py_locations.h"
    click STATUS "https://github.com/google-deepmind/code_contests/blob/main/execution/status_macros.h"
    click STATUS "https://github.com/google-deepmind/code_contests/blob/main/execution/status_matchers.h"
    click TESTS "https://github.com/google-deepmind/code_contests/blob/main/execution/tester_sandboxer_test.cc"
    click WORKSPACE_FILE "https://github.com/google-deepmind/code_contests/tree/main/WORKSPACE"
    click BUILD_TOP "https://github.com/google-deepmind/code_contests/tree/main/BUILD"
    click BUILD_EXEC "https://github.com/google-deepmind/code_contests/blob/main/execution/BUILD.bazel"
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/BUILD.bazel"
    %% Individual third_party build files
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/snappy.BUILD.bazel"
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/zlib.BUILD.bazel"
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/highwayhash.BUILD.bazel"
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/net_zstd.BUILD.bazel"
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/crc32.BUILD.bazel"
    click BUILD_3P "https://github.com/google-deepmind/code_contests/blob/main/third_party/farmhash.BUILD"

    %% Styles
    classDef storage fill:#ADD8E6,stroke:#000;
    classDef libs fill:#90EE90,stroke:#000;
    classDef exec fill:#FFA500,stroke:#000;
    classDef external fill:#D3D3D3,stroke:#000;
```
