How ibalajisivarajan/doomscroll-bias works
This repository implements a preregistered paired-vignette audit of LLM judgments about doomscrolling-related relationship conflict. The main workflow expands frozen vignette variants into model requests, caches responses, scores them, then runs statistical analysis and quality checks. Separate validation and finalization stages support blinded human coding, release readiness, and archival publication; a display-only Notion sync reports study progress. The map follows README capabilities and sampled source; some final release wiring is documented only indirectly, so uncertain orchestration links are omitted.
Main entry points: finalize_study.py Study finalization · run.py Task collection · tables Scored responses · notion_sync.py Dashboard sync
Components 23
Study design
- Study protocol
config/protocol.yaml - Vignette set
data/vignettes.jsonl - Control key
config/control_answer_key.json
Collection
- Task collection
src/run.py - LLM endpoints
- Raw response cache
results/raw - Daily quota counts
results
Scoring and analysis
- Response scoring
src/score.py - Scored responses
results/tables - Statistical analysis
src/analyze.py - Study results
results/tables
Validation and release
- Control QC
src/control_check.py - Validation sampling
src/sample_manual_validation.py - Manual coding files
results/tables - Agreement analysis
src/manual_validation_agreement.py - Study finalization
src/finalize_study.py - Frozen release records
- Zenodo publisher
src/zenodo_release.py
Operations
- Dashboard sync
src/notion_sync.py - Notion dashboard
Other components
- Researcher
- Human coder
- Zenodo archive
Connections 35
- Researcher → Study protocol: defines study
- Researcher → Task collection: starts collection
- Task collection → Study protocol: reads design
- Task collection → Vignette set: loads variants
- Task collection → LLM endpoints: sends prompts
- LLM endpoints → Task collection: returns judgments
- Task collection → Raw response cache: writes responses
- Task collection → Daily quota counts: tracks requests
- Response scoring → Raw response cache: reads responses
- Response scoring → Study protocol: reads scoring rules
- Response scoring → Scored responses: writes scores
- Statistical analysis → Scored responses: reads scores
- Statistical analysis → Study protocol: reads statistics plan
- Statistical analysis → Study results: writes findings
- Control QC → Scored responses: reads scores
- Control QC → Control key: reads criteria
- Control QC → Study results: writes QC summary
- Validation sampling → Scored responses: samples responses
- Validation sampling → Manual coding files: writes worksheet
- Human coder → Manual coding files: adds labels
- Agreement analysis → Manual coding files: reads labels
- Agreement analysis → Frozen release records: writes agreement
- Study finalization → Raw response cache: checks completion
- Study finalization → Response scoring: runs scoring
- Study finalization → Control QC: runs QC
- Study finalization → Validation sampling: creates sample
- Study finalization → Statistical analysis: runs analysis
- Study finalization → Manual coding files: checks coding gate
- Study finalization → Frozen release records: writes manifests
- Dashboard sync → Raw response cache: counts responses
- Dashboard sync → Scored responses: summarizes scores
- Dashboard sync → Study results: reads QC state
- Dashboard sync → Notion dashboard: updates dashboard
- Researcher → Zenodo publisher: publishes release
- Zenodo publisher → Zenodo archive: uploads archive