How bdsaglam/verifiers works
Main Components
• Core RL and Verification Engine:
• Environments (in src/verifiers/envs): e.g., SimpleEnv, MathEnv, ToolEnv, etc.
• Trainers (in src/verifiers/trainers): GRPOEnvTrainer is a primary example that configures and runs the training loop.
• Prompts and Rubrics (in src/verifiers/prompts and src/verifiers/rubrics): Responsible for few-shot prompting, evaluation rubrics, and formatting checks.
• Parsers and Data Utils (in src/verifiers/parsers, src/verifiers/utils, and src/verifiers/datasets): Handle dataset formatting, XML parsing, and other utility functions.
• Tools (in src/verifiers/tools): Include external functionality like the calculator tool.
• Data and Experiment Management:
• “Data” component (represented by data/, dvc.yaml, dvc.lock, and DVC directories) for managing datasets and model checkpoints.
• “Configs” (configs/ directory) that provide YAML files for environment configuration and training (e.g., DeepSpeed ZeRO 3 configuration).
• Orchestration and Workflows:
• “DSTacks” and Docker Compose: Show that there are containerized workflows (dstacks/ and docker-compose.yml) and services for running different parts of the system (under services/). These include:
• Jupyter notebooks for interactive experiments (nbs/)
• Script-based execution flows (scripts/)
• Containerized services for load testing and running inference (in services/, with Dockerfiles and auxiliary scripts)
• External Integrations:
• Logging and Monitoring integrations (e.g., wandb and possibly the usage of gunicorn for serving inference data).
Relationships and Data Flow
• The Environment module (e.g., ToolEnv) which encapsulates the multi-step RL environment.
• The Prompts & Rubrics module, where the few-shot examples and evaluation criteria are applied.
Architectural Patterns and Design Highlights
• Modularity: The project is organized into self-contained modules – environments, tools, prompts, and datasets which can be extended or replaced.
• Configuration-Driven Execution: The use of YAML configuration files (configs/zero3.yaml) and DVC for experiment reproducibility.
• Containerization and Service-Oriented Design: With services, Dockerfiles, and DSTacks configuration, the architecture supports deployment across different environments.
• Multi-step Execution and RL Feedback Loop: Highlight the iterative interaction between LLMs and environments wherein the user-defined multi-step environment (inheriting from MultiStepEnv) feeds back into the training loop.
Detail Level
• Recursive relationships (such as the feedback loop in RL training) and include any auxiliary services that assist in logging, testing (tests/ folder), or model evaluation.