How yurakonoplya/unmark-checker works
unmark-checker is a local benchmark for measuring whether text-rewriting tools remove a keyed SynthID-class watermark, and what meaning, facts, and length are lost. Its main workflow generates or loads a marked sample, accepts a tool’s returned text, rejects unrelated output, scores watermark survival and rewrite costs, then reports the measurements. A separate matrix command aggregates recorded service runs. The graph reflects README-described capabilities and sampled module responsibilities; some CLI wiring and unsampled subsystems are documented but not fully visible in the excerpts.
Main entry points: cli.py CLI commands · kpi.py Rewrite metrics · run.py Remover adapter · detector.py Key-aware detector
Components 17
Sample workflow
- CLI commands
src/unmark_checker/cli.py - Marked samples
samples - Scheme presets
src/unmark_checker/schemes.py - Text generation
src/unmark_checker/watermark.py - Human corpus
src/unmark_checker/corpus.py
Detection and metrics
- Sample identity
src/unmark_checker/identify.py - Key-aware detector
src/unmark_checker/detector.py - Frozen outcomes
src/unmark_checker/thresholds.py - Rewrite metrics
src/unmark_checker/kpi.py - Fact preservation
src/unmark_checker/entities.py - Meaning similarity
src/unmark_checker/embed.py
Results and comparison
- Service records
results/services - Comparison matrix
src/unmark_checker/matrix.py
Tool integration
- Remover adapter
integrations/watermarks-remover/run.py
Other components
- Tester
- Rewrite tool
- Remover service
Connections 20
- Tester → CLI commands: runs commands
- CLI commands → Scheme presets: builds scheme
- CLI commands → Text generation: generates text
- CLI commands → Human corpus: loads control text
- CLI commands → Marked samples: writes samples
- Tester → Rewrite tool: submits sample
- Rewrite tool → CLI commands: returns text
- CLI commands → Sample identity: checks overlap
- CLI commands → Key-aware detector: scores watermark
- Key-aware detector → Text generation: builds processor
- CLI commands → Frozen outcomes: uses outcomes
- CLI commands → Rewrite metrics: measures rewrite
- Rewrite metrics → Fact preservation: checks entities
- Rewrite metrics → Meaning similarity: compares meaning
- CLI commands → Service records: reads records
- CLI commands → Comparison matrix: builds table
- Comparison matrix → Service records: loads runs
- Tester → Remover adapter: runs adapter
- Remover adapter → Remover service: calls service
- Remover adapter → CLI commands: measures returned text