How so1vanlaso1/last_pipeline works
1. Overall shape
- This repo is a competition submission for EXACT 2026 with one serving entrypoint and two distinct QA pipelines behind it.
- The runtime architecture is centered on serve/, which exposes POST /predict and coordinates model servers plus deterministic tooling.
2. Primary serving boundary: serve/
- serve/gateway/app.py is the public HTTP layer.
- It accepts a request shaped as {query_id, type, query, premises, options} and always returns a list of answers.
- The gateway is the orchestration boundary: it does not solve logic/physics itself; it delegates to adapters and model clients.
3. Type 1 path: logic generate→judge
- Logic requests go through serve/gateway/logic_adapter.py.
- The flow is: two generators run concurrently → a judge model reviews them → the gateway emits the final answer, explanation, and premises-used indices.
- The default resident models are Qwen3-4B and Qwen3-4B-Instruct as generators, with Gemma-4-E4B-it as judge.
4. Type 2 path: physics solver-first
- Physics requests go through serve/gateway/physics_adapter.py.
- The flow is: deterministic physics solver first → optional generator stage → judge decides the final answer/unit.
- This path blends symbolic/analytic solving with LLM arbitration rather than relying on the LLM alone.
5. Model-serving layer
- serve/gateway/vllm_client.py talks to vLLM servers using an OpenAI-compatible API.
- serve/run_server.sh and setup.sh launch one vLLM server per resident model, then the gateway on top.
- GET /v1/models aggregates the resident model list across those servers, which is part of the compliance story for the submission.
6. Residency and GPU budget control
- serve/gateway/residency.py manages the sleep/wake behavior that swaps the judge in and out of GPU memory.
- The default design is a swap-based residency scheme: generators stay resident together, and the judge is offloaded/slept when not in use.
- serve/gateway/config.py and serve/logic_config.yaml define the line-up, model roles, quantization, and budget checks.
7. Shared model lineup config
- serve/logic_config.yaml is the canonical deployment config for both logic and physics serving.
- It defines model roles (generator, judge), per-model parameters, quantization, thinking mode, and the total residency budget.
- This config is what determines which models are actually downloaded and launched on the GPU box.
8. Physics pipeline as a standalone subsystem
- physic_pipeline/src/exact_fama/ is the full research-grade physics stack.
- Key internal layers:
- router.py routes tasks.
- physics/ contains domain solvers, formula registry, extraction, and verification.
- logic/ contains symbolic/FOL solving support used when physics questions include logic-like structure.
- llm/ contains parser and fallback/rewrite components.
- pipeline.py and api.py are the orchestration surfaces.
- This subsystem is deterministic-first and can run with or without LLM help.
9. Logic pipeline as a standalone subsystem
- logic_pipeline/src/ is a separate generate→judge experimentation pipeline for type1-style tasks.
- It contains the prompt templates, cascade orchestration, model chat wrapper, scoring, and dataset loading.
- This pipeline can be run independently for offline evaluation and development, but the serving gateway imports the same conceptual behavior.
10. Data, evaluation, and experiment tooling
- physic_pipeline/data/ and logic_pipeline/Logic_Based_Educational_Queries.json hold datasets and benchmark material.
- Both pipelines include scripts for running inference, evaluating outputs, filtering errors, and generating train/eval splits.
- The repo is therefore not just a server; it is also an experimentation and benchmarking workspace.
11. Testing and no-GPU stubs
- serve/tests/ provides wiring tests that use a stub backend instead of GPU models.
- logic_pipeline/tests/ and physic_pipeline/tests/ validate solver behavior, router logic, API schema, and regression cases.
- These tests protect the boundaries between deterministic solvers, model adapters, and the HTTP layer.
12. Operational/deployment concerns
- The serving stack is designed to be launched on a fresh GPU box with a single command and optionally exposed through a Cloudflare tunnel.
- Runtime behavior is controlled by environment variables for model choice, swap behavior, quantization, and ports.
- The gateway is the only public interface; the model servers are internal implementation detail.
13. Important architectural takeaway
- The repo is best drawn as three layers:
1. Public gateway/API
2. Two task pipelines: logic and physics
3. Resident vLLM model servers + deterministic solvers
- The data flow is request → routing by type → pipeline-specific orchestration → model/solver execution → normalized response.
Main entry points: cascade.py Cascade · app.py Gateway API · router.py Router · logic_adapter.py Logic adapter
Components 24
Serving stack
Public gateway, adapters, residency control, and model-client layer for competition serving.
- Gateway API · Public POST /predict boundary that normalizes requests and returns a list of answers.
serve/gateway/app.py - Logic adapter · Type-1 request path: two generators run, then a judge produces the final answer.
serve/gateway/logic_adapter.py - Physics adapter · Type-2 request path: deterministic solver first, optional generation, then judge arbitration.
serve/gateway/physics_adapter.py - Residency · Controls judge sleep/wake behavior and resident model placement under the GPU budget.
serve/gateway/residency.py - vLLM client · Calls resident model servers through an OpenAI-compatible API and aggregates resident models.
serve/gateway/vllm_client.py - Serve config · Runtime settings for roles, ports, quantization, and budget checks.
serve/gateway/config.py - Model lineup · Canonical resident model lineup, roles, and residency budget for serving.
serve/logic_config.yaml - Launch scripts · Shell entrypoints that start one vLLM server per resident model and then the gateway.
serve/run_server.sh
Logic pipeline
Standalone generate-judge pipeline for type-1 tasks and offline experimentation.
- Cascade · Coordinates generation, judging, and answer assembly for logic experiments.
logic_pipeline/src/cascade.py - Chat model · Thin model invocation layer used by the logic pipeline.
logic_pipeline/src/chat_model.py - Prompts · Task prompts and judge/generator instructions for logic workflows.
logic_pipeline/src/prompts.py - Data loader · Loads benchmark data and prepares examples for logic evaluation.
logic_pipeline/src/data_load.py - Scoring · Scores, filters, and ranks candidate answers in offline logic runs.
logic_pipeline/src/score.py
Physics pipeline
Deterministic-first physics subsystem with routing, solvers, parsing, and orchestration.
- Pipeline API · Top-level API surface for physics inference and composition.
physic_pipeline/src/exact_fama/api.py - Router · Dispatches physics questions into solver, parser, and logic-assisted paths.
physic_pipeline/src/exact_fama/router.py - Physics solver · Deterministic physics solving core that uses registered domain solvers.
physic_pipeline/src/exact_fama/physics/solver.py - Solver registry · Registers domain-specific physics solvers and formula resources.
physic_pipeline/src/exact_fama/physics/physics_solvers/registry.py - Logic solver · Handles logic-like structure inside physics problems with symbolic and FOL support.
physic_pipeline/src/exact_fama/logic/solver.py - LLM helpers · Parser, rewrite, and fallback helpers for LLM-assisted physics processing.
physic_pipeline/src/exact_fama/llm/structured_parser.py - Verification · Normalizes and verifies answers, units, and extracted physics results.
physic_pipeline/src/exact_fama/verification.py
Ops and data
Deployment scripts, configs, datasets, and tests supporting serving and evaluation.
- Datasets · Bundled physics datasets and benchmark material used for training and evaluation.
physic_pipeline/data/external_dataset - Logic tests · Wiring and smoke tests that validate the logic pipeline.
logic_pipeline/tests/test_smoke.py - Serve tests · Gateway tests that use a stub backend instead of GPU models.
serve/tests/test_gateway_stub.py - Physics tests · API, router, solver, and upgrade tests for the physics subsystem.
physic_pipeline/tests/test_router.py
Connections 27
- Gateway API → Logic adapter: type=1 · Dispatches logic requests to the generate-judge branch.
- Gateway API → Physics adapter: type=2 · Dispatches physics requests to the solver-first branch.
- Gateway API → Serve config: loads config · Reads runtime settings and model roles.
- Gateway API → Residency: manages GPU · Uses residency control to keep the judge swapped appropriately.
- Logic adapter → vLLM client: generators · Invokes resident generator models via vLLM.
- Logic adapter → Residency: wake/sleep · Brings the judge into GPU memory when needed.
- Physics adapter → vLLM client: judge/gen · Uses resident models for optional generation and final judging.
- Physics adapter → Residency: swap judge · Controls judge residency around physics inference.
- vLLM client → Model lineup: resident list · Reads the canonical model lineup and roles.
- Model lineup → Serve config: budgeting · Defines model roles and residency budget constraints.
- Launch scripts → vLLM client: starts servers · Launches one vLLM server per resident model before the gateway.
- Cascade → Chat model: LLM calls · Wraps model chat for generation and judging.
- Cascade → Prompts: prompting · Uses task-specific prompts for each stage.
- Cascade → Data loader: examples · Consumes loaded benchmark items and splits.
- Cascade → Scoring: rank/filter · Scores candidates and produces final selection.
- Pipeline API → Router: orchestrates · Top-level orchestration for physics inference.
- Router → Physics solver: solve · Sends solvable tasks to deterministic physics solvers.
- Physics solver → Solver registry: domains · Looks up domain-specific formulas and solver modules.
- Router → Logic solver: symbolic path · Routes logic-like physics structure to symbolic solving.
- Router → LLM helpers: fallback parse · Uses parsing and rewrite helpers when LLM assistance is needed.
- Pipeline API → Verification: normalize · Checks and normalizes final answers and units.
- Pipeline API → Datasets: benchmarks · Evaluates against bundled datasets and external packs.
- Physics adapter → Pipeline API: parity · Serving path mirrors the standalone physics pipeline behavior.
- Logic adapter → Cascade: parity · Serving path mirrors the standalone logic pipeline behavior.
- Serve tests → Gateway API: wires · Validates the HTTP gateway with stubbed backends.
- Logic tests → Cascade: regresses · Checks the logic pipeline behavior and smoke paths.
- Physics tests → Router: regresses · Checks router, solver, API schema, and solver upgrades.