How superkevinzhou/hpdg works
hpdg is a Rust library crate centered on deterministic test/data generation for competitive programming and OI workflows. Architecturally, it is organized as a layered generator toolkit: a small foundational core provides shared types and utilities, and the higher-level modules build domain-specific generators and exporters on top of that.
High-level structure
The repository splits into four major surfaces:
1. Core Rust library
- src/lib.rs is the public crate entry point.
- The main functionality lives in src/*.rs modules, which are also exposed through feature flags.
- This is the primary supported API surface.
2. Optional foreign-language bindings
- bindings/c/hpdg.h plus src/c_api.rs provide a C-facing ABI layer.
- bindings/python/lib.rs is the Python binding scaffold.
- The README explicitly says these are present but still early; they are not the primary production surface.
3. Examples, benches, and docs
- examples/full_generation.rs demonstrates end-to-end generation.
- benches/basic.rs exercises performance-sensitive paths.
- docs/*.md contains usage guidance and API notes.
4. Project automation
- .github/workflows/ci.yml runs CI across library, examples, benches, and tests.
Core architectural layers
1) Foundational layer
These modules form the base that other generators depend on:
• core.rs
• traits.rs
• error.rs
• rng.rs
• utils.rs
This layer likely defines:
• shared error handling,
• generator traits and extension points,
• reproducible random-number generation,
• common helper functions,
• base abstractions reused by all domain modules.
The README emphasizes trait-based extension points, seeded RNG streams, and shared error types, so this layer is the stability boundary for the rest of the crate.
2) Domain generator modules
These are the main “payload” of the crate and represent the architecture’s outward-facing subsystems:
• graph.rs
• math.rs
• sequence.rs
• vector.rs
• string.rs
• polygon.rs
• query.rs
• compare.rs
Each module focuses on a specific kind of generated content:
• Graph: graph structures, traversal-friendly formats, graph families, degree-sequence constructors, matrix/list export, and graph-editing helpers.
• Math: number theory and combinatorics helpers.
• Sequence: formula-driven and seeded sequence generation.
• Vector: integer/float vectors and matrix helpers.
• String: random text generation, dictionary-based generation, and lightweight regex-style generation.
• Polygon: points, polygons, convex hull, perimeter/area helpers.
• Query: range queries and mixed update/query streams.
• Compare: file/string/program comparison and grader-style checks.
These modules are not isolated islands; they share the foundational layer for types, RNG, formatting, and errors.
3) I/O and testcase orchestration
These modules handle moving generated data into consumable outputs:
• io.rs
• testcase.rs
This is the “assembly line” of the crate. The README’s feature list makes this clear:
• in-memory buffers,
• configurable naming/separators/formatters,
• flush-to-disk,
• output generation via external programs,
• timeout helpers,
• parallel generation,
• streaming writers,
• batch builders.
This layer sits between the domain generators and the final artifact written to disk or passed to another program. It is the part that turns individual generated objects into complete testcases.
4) Mathematical and geometric support types
Several modules look like reusable support structures rather than standalone generators:
• vector.rs
• graph.rs
• polygon.rs
• sequence.rs
• math.rs
The README mentions Rust-specific building blocks such as:
• NodeId / EdgeId newtypes,
• Weight trait,
• graph generator traits,
• reusable formatting/export helpers.
That suggests a deliberate type-rich design: domain objects are represented with structured types instead of plain tuples, so invariants can be enforced before output generation.
Data flow through the system
The typical flow in this repository is:
1. Seed / initialize RNG
- A reproducible RNG stream is created in the core layer.
2. Build domain objects
- A user calls into a generator module such as graph, vector, string, math, or polygon.
- The module produces typed values or structured collections.
3. Assemble testcase / output
- io and testcase collect generated pieces into buffers or streams.
- Formatting rules, separators, and naming conventions are applied here.
4. Emit artifact
- Output can be flushed to disk.
- In some workflows, output can be piped into an external program and compared or validated.
5. Optionally validate
- compare can compare generated files/strings/program output with custom grading logic.
This makes the architecture a pipeline: core primitives → domain generation → testcase assembly → output/validation.
Important boundaries
Internal vs public API
The library is Rust-first and the Rust API is the main contract. The public surface is organized by feature flags, so consumers can include only the modules they need.
Stable core vs evolving surfaces
The README is explicit that:
• the crate is usable now,
• the public API is still evolving,
• Python/C bindings are scaffolding-level.
So the architecture has a stable center and experimental edges.
Generator logic vs presentation/output
Generation is separated from serialization/formatting:
• domain modules build values and structures,
• I/O modules decide how to render and write them.
This separation matters for architecture diagrams because it keeps generation reusable across multiple output targets.
Rust native vs foreign bindings
The C and Python surfaces are secondary adapters over the Rust core:
• the C header exposes the ABI boundary,
• the Python module wraps or mirrors Rust capabilities,
• neither is the primary implementation layer.
Technology and runtime characteristics that affect architecture
• Language/runtime: Rust library crate.
• Packaging: published to crates.io.
• Build system: Cargo with optional feature flags.
• Parallelism: supported in generation and comparison helpers.
• Determinism: seeded RNG streams are a first-class concern.
• Interop: C ABI and Python binding scaffolding exist.
• Distribution: docs, examples, benches, and CI are part of the repository, indicating the library is intended for practical reuse, not just internal development.
How to draw the architecture
A clear diagram should show:
• A central Rust core library box.
• Beneath it, a foundation layer: RNG, traits, error handling, utilities.
• Above that, several domain generator boxes: graph, math, sequence, vector, string, polygon, query.
• To the side, an I/O / testcase orchestration box that receives generated values and emits files/streams/program input.
• Another side box for comparison / grading utilities.
• External adapter boxes for C and Python bindings connected only to the public library layer.
• Surrounding repository assets for examples, benches, docs, and CI.
The key architectural idea is that hpdg is not a monolithic generator; it is a composable generation framework with a type-safe Rust core, domain-specific generation modules, and separate output/validation adapters.
Main entry points: lib.rs Crate entry · io.rs I/O · ci.yml CI · graph.rs Graph
Components 23
Core library
Public Rust crate entry point and shared foundational modules.
- Crate entry · Main Rust library entry point and feature-gated module surface.
src/lib.rs - Core types · Shared base abstractions used across generators.
src/core.rs - Traits · Generator and integration traits that define composable behavior.
src/traits.rs - RNG · Seeded random-number generation and reproducible streams.
src/rng.rs - Errors · Common error types and result handling for the crate.
src/error.rs - Utils · Reusable helper routines shared by multiple modules.
src/utils.rs
Generators
Domain-specific generators built on the shared core.
- Graph · Graph structures, families, exports, and graph-editing helpers.
src/graph.rs - Math · Number-theory and combinatorics support for generation.
src/math.rs - Sequence · Formula-driven and seeded sequence generation.
src/sequence.rs - Vector · Integer, float, and matrix-oriented vector generation.
src/vector.rs - String · Random text, dictionary-based, and regex-style string generation.
src/string.rs - Polygon · Points, polygons, convex hull, and area/perimeter helpers.
src/polygon.rs - Query · Range queries and mixed update/query streams.
src/query.rs
I/O pipeline
Testcase assembly, output rendering, and comparison utilities.
Bindings
Secondary foreign-language entry points over the Rust core.
- C API · C-facing ABI layer over the Rust library.
src/c_api.rs - Python binding · Python binding scaffold over the Rust core.
bindings/python/lib.rs - C header · C header exposing the public ABI surface.
bindings/c/hpdg.h
Repo surface
Examples, benches, docs, and CI that exercise the library.
- Example · End-to-end generation example demonstrating the crate in action.
examples/full_generation.rs - Bench · Benchmark harness for performance-sensitive generation paths.
benches/basic.rs - Quickstart · Rust usage and API guidance.
docs/rust-quickstart.md - CI · Continuous integration over library, examples, benches, and tests.
.github/workflows/ci.yml
Connections 32
- Crate entry → Core types: re-exports · Public crate surface depends on the shared core.
- Crate entry → Traits: exposes · Traits define the main extension surface.
- Crate entry → RNG: exposes · Deterministic RNG support is part of the core API.
- Crate entry → Errors: exposes · Shared error handling is part of the core API.
- Crate entry → Utils: exposes · Utility helpers are made available through the crate.
- Crate entry → Graph: exports · Graph generation is a primary domain module.
- Crate entry → Math: exports · Math helpers and generators are a primary domain module.
- Crate entry → Sequence: exports · Sequence generation is a primary domain module.
- Crate entry → Vector: exports · Vector generation is a primary domain module.
- Crate entry → String: exports · String generation is a primary domain module.
- Crate entry → Polygon: exports · Polygon and geometry generation are a primary domain module.
- Crate entry → Query: exports · Query workload generation is a primary domain module.
- Graph → Core types: uses · Graph generation builds on shared core types.
- Math → Core types: uses · Math generators use shared core abstractions.
- Sequence → RNG: uses · Sequence generation relies on deterministic RNG streams.
- Vector → Traits: uses · Vector generation follows shared generator traits.
- String → Utils: uses · String generation shares utility routines.
- Polygon → Math: uses · Geometry helpers build on numeric support.
- Query → Vector: uses · Query workloads often compose vector-like data.
- Testcase → I/O: orchestrates · Testcase assembly feeds the output pipeline.
- I/O → Graph: renders · I/O layer formats generated domain objects.
- I/O → String: renders · I/O layer formats generated domain objects.
- Compare → I/O: consumes · Comparison utilities operate on emitted artifacts.
- C API → Crate entry: wraps · C ABI is an adapter over the Rust library.
- Python binding → Crate entry: wraps · Python bindings sit on top of the Rust core.
- C header → C API: declares · The header matches the C-facing API layer.
- Example → Crate entry: demonstrates · Example exercises the primary Rust API.
- Bench → Crate entry: measures · Benchmarks target the crate's hot paths.
- Quickstart → Crate entry: documents · Docs explain how to use the Rust API.
- CI → Crate entry: tests · CI validates the library and its surfaces.
- CI → Example: runs · CI includes example execution.
- CI → Bench: checks · CI includes benchmark-related validation.