How google-deepmind/code_contests works
Project type and purpose
• Type: A data-processing and execution library/toolkit for competitive programming datasets.
• Purpose:
– Provide protocol-buffer definitions and I/O (via Riegeli) for contest problems.
– Offer C++ and Python example clients to iterate over the dataset.
– Supply an execution engine (sandboxed) to compile and run submitted solutions for correctness/evaluation.
File structure
• Top-level:
– BUILD files and WORKSPACE: Bazel as build orchestrator
– contest_problem.proto: central data schema
– print_names.cc / print_names_and_sources.py: example ingestion clients
– execution/: C++ sandboxed execution engine (BUILD.bazel, solver, sandboxer, threadpool, tests)
– third_party/: external compression, hashing, and I/O libraries (snappy, zlib, highwayhash, net_zstd, crc32, farmhash)
• Patterns:
– Modular libraries (ingestion vs. execution)
– Clear separation of data schema, I/O, execution, and third-party dependencies.
README for technical stack
• Languages: C++17, Python 2.7/3.x
• Build: Bazel on Linux (clang)
• Data format: Riegeli-encoded protocol buffers
• Execution: sandboxing (resource-limited child processes) via tester_sandboxer
• Dependencies: Cloud SDK (gsutil) for data download
a. Data Storage
– GCS bucket “dm-code_contests”
– .riegeli files (train/valid/test)
b. Data Schema & I/O Library
– contest_problem.proto → Bazel-built C++/Python bindings
– Riegeli reader/writer (third_party/riegeli)
– Example clients:
• print_names.cc (C++)
• print_names_and_sources.py (Python)
c. Execution Engine
– solve_example binary → CLI front-end
– Execution Controller (solve_example, py_tester_sandboxer)
– Thread Pool (simple_threadpool.h)
– Sandboxer modules (tester_sandboxer.cc/h, py_tester_sandboxer.cc/h)
– Python interpreters (paths from py_locations.cc)
– Status/reporting (status_macros.h, status_matchers.h)
d. Build System & Toolchain
– Bazel WORKSPACE & BUILD.bazel files
– External dependencies under third_party
e. External Tools & Environments
– Cloud SDK (gsutil) for data transfer
– System interpreters: /usr/bin/python3.9, /usr/bin/python2.7
Relationships and data-flow
1. GCS Bucket → Data Downloader (gsutil cp) → Local Riegeli files
2. Riegeli files → Example clients (print_names, solve_example)
3. Example clients → I/O Library → contest_problem proto objects
4. solve_example → Execution Engine → Sandboxer → invokes Python processes
5. Sandboxer → returns status/results to solve_example → stdout/log
• Build-time dependencies:
– Bazel orchestrates compilation of ingestion clients, execution engine, and links third-party libs.
Architectural patterns & design principles
• Modular layering: Data Schema → I/O → Processing/Execution → Client CLI
• Separation of concerns: ingestion vs. execution vs. tooling
• Pluggable sandbox: both C++ and Python sandbox implementations
• Testability: separate tester_sandboxer_test.cc to validate sandbox behavior
• Clear component labels and boundaries
• Version or platform note (“Linux/clang, Python2.7 & 3.x”)