devlup-labs/autosetter

Purpose and entry point

AutoSetter converts a problem-statement image or PDF into a reviewable Codeforces/Polygon-style package: specification, statement, C++ solution/validator/generator/checker, accepted tests, validation report, and manifest. The CLI entry point is app.py; its output pipeline is represented concretely under example/01_intake through example/04_package.

1. Intake and specification extraction

app.py accepts an image/PDF and produces the canonical problem.json specification. Image/PDF handling and structured-model-output processing are separated into utils/image_parser.py and utils/json_generator.py. The extraction prompt is prompts/json_extraction.txt; example/01_intake/problem.json shows the handoff format.

2. Local model boundary

All generation is driven by local Qwen models exposed through Ollama, isolated behind utils/ollama_client.py. Prompt selection/loading is centralized in utils/prompt_loader.py, with artifact-specific contracts in prompts/statement.txt, prompts/solution.txt, prompts/validator.txt, prompts/generator.txt, and prompts/checker.txt. This boundary allows most pipeline logic to run without a model service.

3. Artifact generation

The normalized JSON spec is faned out into independently generated deliverables: Markdown statement plus four C++ programs. File materialization is handled by utils/file_generator.py; the expected generated set is illustrated by example/02_build/statement.md, example/02_build/solution.cpp, and example/02_build/validator.cpp. Generated C++ is compiled against the vendored third_party/testlib.h.

4. Test planning and validation pipeline

utils/test_plan.py defines test-shape intent (boundary, structural, and multi-case distributions), rather than relying only on random seeds. utils/test_pipeline.py performs the critical control flow: compile artifacts; verify validator acceptance of statement samples; generate candidates; reject validator-invalid inputs; solve accepted inputs; and run the checker on reference answers. Outputs include numbered .in/.ans pairs and a diagnosis-bearing report, as shown in example/03_validate/validation_report.json.

5. Independent correctness gates

Release readiness is deliberately stricter than successful generation. The validator is checked against problem samples and against generated tests. The checker must reject deliberately bad outputs (empty, truncated, perturbed), preventing a checker that merely accepts matching files. Test-plan conformance checks generator behavior, including deterministic minimum-case generation. These gates determine whether the process exits successfully or returns the “not fit to release” status.

6. Packaging boundary

utils/packager.py converts accepted artifacts and tests into the portable package layout: problem metadata, statement, source files, solutions, tests, report, and manifest. example/04_package/manifest.json, example/04_package/problem.json, and example/04_package/tests demonstrate the release bundle. The manifest’s readiness flag and validation report are the human-review decision surface.

7. Execution infrastructure

Python is the orchestration runtime; generated contestants/tooling are C++ compiled with g++. A separate Node.js sandbox service executes untrusted C++ inside Docker/nsjail: sandbox/server/src/server.js, sandbox/server/src/executor.js, and sandbox/server/src/config.js. Docker construction is in sandbox/docker/Dockerfile; configuration enforces CPU, memory, PID, and network limits. Ollama is the only required external model service.