# ericckzhou/falsifyai architecture

> Architecture diagram of the GitHub repository ericckzhou/falsifyai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ericckzhou/falsifyai
- Repository: https://github.com/ericckzhou/falsifyai
- Video tour (about a minute): https://gitdiagram.com/ericckzhou/falsifyai/video

## Overview

FalsifyAI is a CLI for pressure-testing LLM workflows and retaining replayable evidence. The principal workflow loads a YAML spec, materializes perturbed inputs, executes model requests, judges outputs, resolves verdicts, and stores a session that can later be inspected or compared. Other documented capabilities include longitudinal views, integrity verification, and portable evidence export. Paths and core execution details are supported by the tree and sampled source; some command-to-subsystem wiring is inferred from the README and is not shown in the excerpts.

Main entry points: [`falsifyai/cli/main.py`](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/main.py) (CLI dispatch), [`falsifyai/replay/protocol.py`](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/protocol.py) (Replay store), [`falsifyai/spec/materializer.py`](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/spec/materializer.py) (Input materialization), [`falsifyai/cli/export.py`](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/export.py) (Evidence export)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Eval workflow"]
  node_cli["CLI dispatch<br/>[main.py]"]
  node_spec["Spec loading<br/>[loader.py]"]
  node_materializer["Input materialization<br/>[materializer.py]"]
  node_perturbations["Perturbation families<br/>[registry.py]"]
  node_paraphrase["Paraphrase validity<br/>[paraphrase.py]"]
  node_engine["Execution engine<br/>[engine.py]"]
  node_adapter["LiteLLM adapter<br/>[litellm_adapter.py]"]
end

subgraph group_evaluation["Judgment"]
  node_invariants["Output invariants<br/>[registry.py]"]
  node_oracles["Semantic oracles<br/>[nli.py]"]
  node_resolver["Verdict resolution<br/>[resolver.py]"]
  node_score["Falsifiability score<br/>[score.py]"]
end

subgraph group_evidence["Replay evidence"]
  node_artifact["Replay artifact<br/>[models.py]"]
  node_serialization["Artifact serialization<br/>[serialize.py]"]
  node_store["Replay store<br/>[protocol.py]"]
  node_sqlite[("SQLite backend<br/>[sqlite_store.py]")]
  node_integrity["Integrity checks<br/>[checks.py]"]
  node_bundle["Evidence bundle<br/>[writer.py]"]
end

subgraph group_review["Review tools"]
  node_readcommands["Replay inspection<br/>[inspect.py]"]
  node_comparisons["Session comparison<br/>[diff.py]"]
  node_trends["History and trends<br/>[timeline.py]"]
  node_matrix["Model matrix<br/>[matrix.py]"]
  node_verify["Artifact verification<br/>[verify.py]"]
  node_export["Evidence export<br/>[export.py]"]
end

node_user(("CLI user"))
node_llm{{"LLM providers"}}

node_user -->|"invokes"| node_cli
node_cli -->|"loads spec"| node_spec
node_cli -->|"runs evaluation"| node_materializer
node_materializer -->|"realizes inputs"| node_perturbations
node_materializer -->|"builds paraphrases"| node_paraphrase
node_paraphrase -->|"requests rewrites"| node_adapter
node_materializer -->|"executes cases"| node_engine
node_engine -->|"executes request"| node_adapter
node_adapter -->|"calls model"| node_llm
node_cli -->|"judges outputs"| node_invariants
node_cli -.->|"uses optional NLI"| node_oracles
node_cli -->|"resolves verdicts"| node_resolver
node_resolver -->|"computes score"| node_score
node_cli -->|"assembles session"| node_artifact
node_artifact -->|"serializes"| node_serialization
node_cli -->|"persists session"| node_store
node_store -->|"stores records"| node_sqlite
node_cli -->|"dispatches inspect"| node_readcommands
node_cli -->|"dispatches diff"| node_comparisons
node_cli -->|"dispatches history"| node_trends
node_cli -->|"dispatches matrix"| node_matrix
node_cli -->|"dispatches verify"| node_verify
node_cli -->|"dispatches export"| node_export
node_readcommands -->|"loads sessions"| node_store
node_comparisons -->|"loads sessions"| node_store
node_trends -->|"loads sessions"| node_store
node_matrix -->|"loads sessions"| node_store
node_verify -->|"loads artifact"| node_store
node_verify -->|"checks integrity"| node_integrity
node_export -->|"loads artifact"| node_store
node_export -->|"checks integrity"| node_integrity
node_export -->|"writes bundle"| node_bundle

click node_cli "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/main.py"
click node_spec "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/spec/loader.py"
click node_materializer "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/spec/materializer.py"
click node_perturbations "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/perturbation/registry.py"
click node_paraphrase "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/perturbation/paraphrase.py"
click node_engine "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/execution/engine.py"
click node_adapter "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/execution/litellm_adapter.py"
click node_invariants "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/invariants/registry.py"
click node_oracles "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/oracles/nli.py"
click node_resolver "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/verdict/resolver.py"
click node_score "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/falsifiability/score.py"
click node_artifact "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/models.py"
click node_serialization "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/serialize.py"
click node_store "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/protocol.py"
click node_sqlite "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/sqlite_store.py"
click node_integrity "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/integrity/checks.py"
click node_bundle "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/bundle/writer.py"
click node_readcommands "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/inspect.py"
click node_comparisons "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/diff.py"
click node_trends "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/timeline.py"
click node_matrix "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/matrix.py"
click node_verify "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/verify.py"
click node_export "https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/export.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_spec,node_materializer,node_perturbations,node_paraphrase,node_engine,node_adapter,node_user toneBlue
class node_invariants,node_oracles,node_resolver,node_score toneAmber
class node_artifact,node_serialization,node_store,node_sqlite,node_integrity,node_bundle toneMint
class node_readcommands,node_comparisons,node_trends,node_matrix,node_verify,node_export toneRose
class node_llm toneIndigo
```

## Components

### Eval workflow

- [CLI dispatch](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/main.py): `falsifyai/cli/main.py`
- [Spec loading](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/spec/loader.py): `falsifyai/spec/loader.py`
- [Input materialization](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/spec/materializer.py): `falsifyai/spec/materializer.py`
- [Perturbation families](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/perturbation/registry.py): `falsifyai/perturbation/registry.py`
- [Paraphrase validity](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/perturbation/paraphrase.py): `falsifyai/perturbation/paraphrase.py`
- [Execution engine](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/execution/engine.py): `falsifyai/execution/engine.py`
- [LiteLLM adapter](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/execution/litellm_adapter.py): `falsifyai/execution/litellm_adapter.py`

### Judgment

- [Output invariants](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/invariants/registry.py): `falsifyai/invariants/registry.py`
- [Semantic oracles](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/oracles/nli.py): `falsifyai/oracles/nli.py`
- [Verdict resolution](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/verdict/resolver.py): `falsifyai/verdict/resolver.py`
- [Falsifiability score](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/falsifiability/score.py): `falsifyai/falsifiability/score.py`

### Replay evidence

- [Replay artifact](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/models.py): `falsifyai/replay/models.py`
- [Artifact serialization](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/serialize.py): `falsifyai/replay/serialize.py`
- [Replay store](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/protocol.py): `falsifyai/replay/protocol.py`
- [SQLite backend](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/replay/sqlite_store.py): `falsifyai/replay/sqlite_store.py`
- [Integrity checks](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/integrity/checks.py): `falsifyai/integrity/checks.py`
- [Evidence bundle](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/bundle/writer.py): `falsifyai/bundle/writer.py`

### Review tools

- [Replay inspection](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/inspect.py): `falsifyai/cli/inspect.py`
- [Session comparison](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/diff.py): `falsifyai/cli/diff.py`
- [History and trends](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/timeline.py): `falsifyai/cli/timeline.py`
- [Model matrix](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/matrix.py): `falsifyai/cli/matrix.py`
- [Artifact verification](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/verify.py): `falsifyai/cli/verify.py`
- [Evidence export](https://github.com/ericckzhou/falsifyai/blob/main/falsifyai/cli/export.py): `falsifyai/cli/export.py`

### Other components

- **CLI user**
- **LLM providers**

## Connections

- CLI user → CLI dispatch: invokes
- CLI dispatch → Spec loading: loads spec
- CLI dispatch → Input materialization: runs evaluation
- Input materialization → Perturbation families: realizes inputs
- Input materialization → Paraphrase validity: builds paraphrases
- Paraphrase validity → LiteLLM adapter: requests rewrites
- Input materialization → Execution engine: executes cases
- Execution engine → LiteLLM adapter: executes request
- LiteLLM adapter → LLM providers: calls model
- CLI dispatch → Output invariants: judges outputs
- CLI dispatch → Semantic oracles: uses optional NLI
- CLI dispatch → Verdict resolution: resolves verdicts
- Verdict resolution → Falsifiability score: computes score
- CLI dispatch → Replay artifact: assembles session
- Replay artifact → Artifact serialization: serializes
- CLI dispatch → Replay store: persists session
- Replay store → SQLite backend: stores records
- CLI dispatch → Replay inspection: dispatches inspect
- CLI dispatch → Session comparison: dispatches diff
- CLI dispatch → History and trends: dispatches history
- CLI dispatch → Model matrix: dispatches matrix
- CLI dispatch → Artifact verification: dispatches verify
- CLI dispatch → Evidence export: dispatches export
- Replay inspection → Replay store: loads sessions
- Session comparison → Replay store: loads sessions
- History and trends → Replay store: loads sessions
- Model matrix → Replay store: loads sessions
- Artifact verification → Replay store: loads artifact
- Artifact verification → Integrity checks: checks integrity
- Evidence export → Replay store: loads artifact
- Evidence export → Integrity checks: checks integrity
- Evidence export → Evidence bundle: writes bundle