How apolloresearch/deception-detection works
Main Components
• Data Management:
• Data Storage: Represent the “data” directory that holds prompts, rollouts, trajectories, and training examples.
• Data Processing Modules: Highlight the “deception_detection/data” package where various dataset subclasses are implemented to process raw files.
• Core Experiment Modules:
• Experimental Orchestration & Configuration: Include the “deception_detection/experiment.py” as the central driver of experiments and its companion config files in “deception_detection/scripts/configs”.
• Experiment Entrypoints: Also note the experiment script at “deception_detection/scripts/experiment.py”.
• Detection & Probing Components:
• Detector Modules: Identify “deception_detection/detectors.py”, “deception_detection/models.py”, and “deception_detection/sae.py” as components for running linear probes and related detection mechanisms.
• Metrics and Prompt Scoring: Include “deception_detection/metrics.py” and “deception_detection/prompt_scorer.py” that evaluate system performance.
• Results & Visualization:
• Results Processing: Note the “deception_detection/results” modules for generating visualizations, plots, and HTML outputs.
• Notebooks and Analysis: Use the “deception_detection/notebooks” folder that contains exploratory analysis and debugging scripts.
• Utility & Integration:
• Infrastructure & Logging: The “deception_detection/log.py”, “deception_detection/utils.py”, and files like “pyproject.toml”, “Makefile”, and CI/CD configurations in “.github” reflect standard development utilities.
• External API Integration: The README and .env recommendations show dependencies on external APIs (e.g., OpenAI, Hugging Face, and Goodfire).
Relationships and Interactions
• Data flows from the raw files in the “data” directory into the dataset loaders implemented in “deception_detection/data”.
• The experiment module orchestrates the processing and evaluation, invoking the detector and probe components to run experiments based on the datasets.
• Detectors use computed metrics, logs, and prompts (from both configurations and generated rollouts) to assess model deception.
• Experiment results pass output into the results and visualization modules, which then produce plots, tables, and reports saved under folders like “example_results” and “deception_detection/results”.
• Supporting scripts under “deception_detection/scripts” act as the glue to perform tasks such as generating rollouts, converting facts to Q&A, or running analysis pipelines.
Architectural Patterns and Design Principles
• Modularity: The clear separation of concerns is evident—data ingestion, experiment orchestration, detection logic, and result visualization are each in separate modules/packages.
• Configurability: The use of YAML config files (in “deception_detection/scripts/configs”) and environment variables for API keys implies an emphasis on configurable experiment parameters.
• Extensibility: The presence of multiple dataset-specific files and generation scripts (for roleplaying, insider trading, etc.) shows that the codebase is designed to plug in new conditions or experiment types.
• Reusability: Utility modules (logging, metrics, and shared functions) help in avoiding code duplication throughout the experimental workflows.