How ericckzhou/falsifyai works
FalsifyAI is a CLI for pressure-testing LLM workflows and retaining replayable evidence. The principal workflow loads a YAML spec, materializes perturbed inputs, executes model requests, judges outputs, resolves verdicts, and stores a session that can later be inspected or compared. Other documented capabilities include longitudinal views, integrity verification, and portable evidence export. Paths and core execution details are supported by the tree and sampled source; some command-to-subsystem wiring is inferred from the README and is not shown in the excerpts.
Main entry points: main.py CLI dispatch · protocol.py Replay store · materializer.py Input materialization · export.py Evidence export
Components 25
Eval workflow
- CLI dispatch
falsifyai/cli/main.py - Spec loading
falsifyai/spec/loader.py - Input materialization
falsifyai/spec/materializer.py - Perturbation families
falsifyai/perturbation/registry.py - Paraphrase validity
falsifyai/perturbation/paraphrase.py - Execution engine
falsifyai/execution/engine.py - LiteLLM adapter
falsifyai/execution/litellm_adapter.py
Judgment
- Output invariants
falsifyai/invariants/registry.py - Semantic oracles
falsifyai/oracles/nli.py - Verdict resolution
falsifyai/verdict/resolver.py - Falsifiability score
falsifyai/falsifiability/score.py
Replay evidence
- Replay artifact
falsifyai/replay/models.py - Artifact serialization
falsifyai/replay/serialize.py - Replay store
falsifyai/replay/protocol.py - SQLite backend
falsifyai/replay/sqlite_store.py - Integrity checks
falsifyai/integrity/checks.py - Evidence bundle
falsifyai/bundle/writer.py
Review tools
- Replay inspection
falsifyai/cli/inspect.py - Session comparison
falsifyai/cli/diff.py - History and trends
falsifyai/cli/timeline.py - Model matrix
falsifyai/cli/matrix.py - Artifact verification
falsifyai/cli/verify.py - Evidence export
falsifyai/cli/export.py
Other components
- CLI user
- LLM providers
Connections 32
- CLI user → CLI dispatch: invokes
- CLI dispatch → Spec loading: loads spec
- CLI dispatch → Input materialization: runs evaluation
- Input materialization → Perturbation families: realizes inputs
- Input materialization → Paraphrase validity: builds paraphrases
- Paraphrase validity → LiteLLM adapter: requests rewrites
- Input materialization → Execution engine: executes cases
- Execution engine → LiteLLM adapter: executes request
- LiteLLM adapter → LLM providers: calls model
- CLI dispatch → Output invariants: judges outputs
- CLI dispatch → Semantic oracles: uses optional NLI
- CLI dispatch → Verdict resolution: resolves verdicts
- Verdict resolution → Falsifiability score: computes score
- CLI dispatch → Replay artifact: assembles session
- Replay artifact → Artifact serialization: serializes
- CLI dispatch → Replay store: persists session
- Replay store → SQLite backend: stores records
- CLI dispatch → Replay inspection: dispatches inspect
- CLI dispatch → Session comparison: dispatches diff
- CLI dispatch → History and trends: dispatches history
- CLI dispatch → Model matrix: dispatches matrix
- CLI dispatch → Artifact verification: dispatches verify
- CLI dispatch → Evidence export: dispatches export
- Replay inspection → Replay store: loads sessions
- Session comparison → Replay store: loads sessions
- History and trends → Replay store: loads sessions
- Model matrix → Replay store: loads sessions
- Artifact verification → Replay store: loads artifact
- Artifact verification → Integrity checks: checks integrity
- Evidence export → Replay store: loads artifact
- Evidence export → Integrity checks: checks integrity
- Evidence export → Evidence bundle: writes bundle