How dodona-edu/universal-judge works
TESTed is a language-agnostic educational judge: an exercise’s configuration, test suite and submission are evaluated, then feedback is emitted in Dodona’s JSON Lines format. The documented workflow enters through the command-line tool; the judge coordinates planning, code generation, compilation/execution and oracle-based evaluation. Supporting branches cover suite DSL translation, language adapters, descriptions, linting and internationalization. The README and source index establish the product capabilities and module references; unsampled implementation details are represented only where those references support the relationship.
Main entry points: testsuite.py Test suite model · core.py Judge orchestration · generation.py Test code generation · main.py Runtime entry
Components 26
Inputs and DSL
- CLI entry
tested/cli.py - Configuration
tested/configs.py - Test suite model
tested/testsuite.py - Suite DSL parser
tested/dsl/translate_parser.py
Judging workflow
- Runtime entry
tested/main.py - Judge orchestration
tested/judge/core.py - Execution planning
tested/judge/planning.py - Compilation
tested/judge/compilation.py - Submission execution
tested/judge/execution.py
Language support
- Input preparation
tested/languages/preparation.py - Test code generation
tested/languages/generation.py - Language adapters
tested/languages - Submission adaptation
tested/languages/conventionalize.py
Evaluation and feedback
- Output oracles
tested/oracles - Result evaluation
tested/judge/evaluation.py - Feedback collection
tested/judge/collector.py - Dodona output format
tested/dodona.py - Test descriptions
tested/descriptions
Shared test model
- Feature analysis
tested/features.py - Data types
tested/datatypes - Serialization model
tested/serialisation.py - Input parsing
tested/parsing.py
Other components
- Exercise author
- Submission
- Language toolchain
- JSONL feedback
Connections 31
- Exercise author → CLI entry: runs judge
- CLI entry → Runtime entry: starts runtime
- Runtime entry → Configuration: loads config
tested/main.py - Runtime entry → Test suite model: loads suite
tested/main.py - Exercise author → Test suite model: defines tests
- Test suite model → Suite DSL parser: translates suite
tested/dsl/translate_parser.py - Judge orchestration → Configuration: uses settings
tested/judge/core.py - Judge orchestration → Test suite model: uses cases
tested/judge/core.py - Judge orchestration → Execution planning: plans units
tested/judge/core.py - Judge orchestration → Test code generation: generates tests
tested/judge/core.py - Judge orchestration → Compilation: compiles code
tested/judge/core.py - Judge orchestration → Submission execution: runs submission
tested/judge/core.py - Judge orchestration → Result evaluation: evaluates results
tested/judge/core.py - Judge orchestration → Feedback collection: collects feedback
tested/judge/core.py - Execution planning → Test suite model: plans from cases
tested/judge/planning.py - Test code generation → Input preparation: prepares cases
tested/languages/generation.py - Test code generation → Submission adaptation: adapts names
tested/languages/generation.py - Test code generation → Language adapters: generates per language
tested/languages/generation.py - Submission → Submission execution: is executed
- Compilation → Language toolchain: uses toolchain
- Submission execution → Language toolchain: invokes toolchain
- Result evaluation → Output oracles: checks outputs
tested/judge/evaluation.py - Result evaluation → Feedback collection: emits updates
tested/judge/evaluation.py - Feedback collection → Dodona output format: formats updates
tested/judge/collector.py - Dodona output format → JSONL feedback: serializes output
- Test descriptions → Test suite model: renders case context
tested/descriptions/renderer.py - Test code generation → Feature analysis: uses capabilities
tested/languages/generation.py - Test suite model → Data types: models values
tested/testsuite.py - Test suite model → Serialization model: uses format model
tested/testsuite.py - Input parsing → Serialization model: parses values
tested/parsing.py - Input parsing → Test suite model: loads suite data
tested/parsing.py