How choeyunbeom/skill-curator works
skill-curator is a flat research harness for safely pruning an append-only, file-based skill library and measuring the effects. The main workflow collects skills, probes them for behavioral duplicates and health on evaluation and drift data, applies lifecycle transitions, then compares curated and raw libraries in benchmark evaluation and replays removals for deletion regret. Separate analysis scripts examine prompt pollution and other results. The README documents these capabilities; some implementation links are shown only where the excerpts or index support them.
Main entry points: run_experiment.py Experiment runner · evaluator.py Benchmark evaluator · sandbox.py Skill sandbox · analyze_flips.py Flip analysis
Components 23
Curation
- Corpus collection
harness/collect_corpus.py - Experiment runner
harness/run_experiment.py - Skill model
harness/modules/skill_model.py - Behavioral dedup
harness/modules/dedup_engine.py - Health probes
harness/modules/task_stream.py - Health monitor
harness/modules/health_monitor.py - Lifecycle manager
harness/modules/lifecycle_manager.py - Regret tracker
harness/modules/regret_tracker.py - Skill sandbox
harness/modules/sandbox.py - Skill runner
harness/modules/_skill_runner.py
Skill library
- Raw library
harness/skill_library/raw - Curated library
harness/skill_library/curated - Recoverable archive
harness/skill_library/archived - Drift data
harness/skill_library/drift_data
Evaluation
- Lexical retrieval
harness/modules/retrieval.py - Code executor
harness/modules/code_executor.py - Benchmark evaluator
harness/modules/evaluator.py
Analysis
- Pollution analysis
harness/analyze_pollution.py - Flip analysis
harness/analyze_flips.py
Other components
- Research operator
- Skill generator
- DABench
- Generator LLM session
Connections 22
- Research operator → Experiment runner: runs
README.md - Corpus collection → Skill generator: harvests from
README.md - Experiment runner → Raw library: loads baseline
README.md - Experiment runner → Curated library: curates copy
README.md - Experiment runner → Behavioral dedup: invokes
harness/run_experiment.py - Experiment runner → Health probes: builds probes
harness/run_experiment.py - Experiment runner → Health monitor: checks health
harness/run_experiment.py - Experiment runner → Lifecycle manager: applies transitions
harness/run_experiment.py - Experiment runner → Regret tracker: records removals
harness/run_experiment.py - Experiment runner → Benchmark evaluator: runs benchmark
harness/run_experiment.py - Behavioral dedup → Skill sandbox: executes probes
harness/modules/dedup_engine.py - Health monitor → Skill sandbox: runs skills
harness/modules/health_monitor.py - Skill sandbox → Skill runner: launches subprocess
harness/modules/sandbox.py - Lifecycle manager → Recoverable archive: moves archived skills
harness/modules/lifecycle_manager.py - Health probes → Drift data: probes drift data
README.md - Benchmark evaluator → Lexical retrieval: retrieves top-k
harness/modules/evaluator.py - Benchmark evaluator → Code executor: executes solver code
harness/modules/evaluator.py - Benchmark evaluator → DABench: evaluates tasks
README.md - Benchmark evaluator → Generator LLM session: asks session
README.md - Pollution analysis → Experiment runner: reuses curation
harness/analyze_pollution.py - Flip analysis → Experiment runner: rebuilds curated set
harness/analyze_flips.py - Flip analysis → Benchmark evaluator: compares task results
harness/analyze_flips.py