How sushit-prog/langgraph-replay works
1. This repo is effectively two related layers: the main shipped package in src/langgraph_replay, and a second agenttrace package with similar-but-separate analysis tooling. The README and install surface point to langgraph_replay as the primary product; agenttrace looks like an experimental or adjacent codebase kept in the same repo for loop detection, counterfactual replay, annotations, and watchdog experiments.
2. The core runtime path is recording LangGraph agent execution through a LangChain callback handler. LangGraphRecorder / record_session / arecord_session wrap graph.invoke(...) or graph.ainvoke(...) and capture node-by-node state transitions plus LLM metadata. This is the ingestion boundary: the agent emits execution events, and the recorder turns them into persisted session data.
3. Persisted sessions are stored locally in SQLite under ~/.langgraph_replay/replays.db. The storage layer is the main durable boundary between “runtime capture” and “offline analysis.” Everything else in the toolchain reads from that session database to list runs, show details, export/delete sessions, compute leaderboards, and replay or compare executions.
4. The analysis pipeline in src/langgraph_replay is split into focused modules:
- replay.py reconstructs a run from recorded events.
- diff.py compares sessions.
- blame.py identifies the node most likely responsible for failure.
- diagnosis.py adds higher-level root-cause reasoning and fix suggestions.
- search.py provides semantic lookup across recorded sessions.
- storage.py and search.py are the query backbone for all of these.
5. “Blame” has two distinct modes. Structural blame is immediate and local: it reasons from state changes and node boundaries without external calls. Semantic blame and diagnosis are optional higher-cost paths that can use model-based evaluation or explanation, which is why the README mentions --eval, --baseline, and --diagnose. That creates a clear boundary between deterministic replay analysis and LLM-assisted interpretation.
6. Semantic search is a separate subsystem rather than a simple text filter. The repository includes a search engine over session content, with embeddings implied by the search API and the provider leaderboard. This means the SQLite store is likely used for both structural metadata and vector/semantic lookup, while the query layer turns natural-language prompts into ranked historical sessions.
7. The regression watchdog is another major flow: watchdog.py and the watch CLI monitor an agent source file, rerun saved sessions after edits, and compare results against previous behavior. The watchdog boundary is “code changed -> replay sessions -> detect regressions -> emit report.” It is intentionally file-driven and local, aimed at catching behavior drift during development rather than in production.
8. The terminal UI is a separate presentation layer for interactive inspection. tui/app.py and tui/widgets.py provide the debug experience invoked by langgraph-replay debug <session>. This sits on top of the same stored sessions and replay logic, so the TUI is a view layer, not a source of truth.
9. The repo also contains a substantial agenttrace package with its own internal subsystems:
- annotations for storing labeled execution metadata,
- counterfactual for replaying alternate paths,
- loopdetect for cycle finding, classification, and embeddings,
- watchdog for baseline comparison, semantic diffing, upstream divergence, and reporting.
These modules suggest a parallel analysis framework focused on detecting loops, retries, regressions, and divergence across runs.
10. Fixtures and tests are architecture-significant here because they encode the expected shapes of failures and edge cases. The various JSON fixtures for clean runs, regressed runs, missing nodes, stuck loops, retries, and upstream divergence act as canonical examples for replay, blame, diff, and watchdog behavior. The tests in tests/ and agenttrace/tests/ validate the boundaries between recording, storage, analysis, and reporting.
11. There are also demo and example entry points that show how the library is embedded into a real agent workflow. examples/simple_agent.py and demo/research_agent.py / demo/run_demo.py demonstrate the recording boundary from the agent side, while generated reports such as demo_report.json and watchdog-report.json show the downstream analysis artifacts.
12. In diagram terms, the main architecture is: LangGraph agent execution enters through a callback recorder, gets persisted to local SQLite, and is then consumed by several read-side tools—list/show/export, replay/diff, blame/diagnosis, semantic search, watchdog, and TUI debugging. Optional model-based features and embeddings introduce external service dependencies, but the core structural analysis remains local and offline.
Main entry points: storage.py Storage · cli.py CLI · watchdog.py Watchdog · clean_run_fixture.json Run fixtures
Components 18
langgraph_replay
Primary shipped package for recording, persistence, replay, analysis, and interactive inspection.
- Recorder · LangChain callback wrapper that captures LangGraph execution events and session metadata.
src/langgraph_replay/recorder.py - Storage · Durable session store, centered on the local replay database used by read-side tools.
src/langgraph_replay/storage.py - Replay · Rebuilds recorded executions from persisted events.
src/langgraph_replay/replay.py - Diff · Compares sessions and highlights behavioral differences.
src/langgraph_replay/diff.py - Blame · Identifies the node most likely responsible for a failure, using structural analysis first.
src/langgraph_replay/blame.py - Diagnosis · Higher-level explanation and fix suggestion layer built on top of blame and replay outputs.
src/langgraph_replay/diagnosis.py - Search · Query and ranking layer for finding relevant historical sessions.
src/langgraph_replay/search.py - CLI · User-facing commands for record, list, show, replay, diff, blame, diagnose, search, and watch flows.
src/langgraph_replay/cli.py - TUI · Terminal debug interface for inspecting stored sessions and replay state.
src/langgraph_replay/tui/app.py - Watchdog · Local watch flow that reruns saved sessions after edits and compares behavior for regressions.
src/langgraph_replay/watchdog.py
agenttrace
Adjacent analysis toolkit in the same repo for annotations, counterfactual replay, loop detection, and watchdog experiments.
- Annotations · Stores execution labels and annotation metadata for analysis and review.
agenttrace/annotations/store.py - Counterfactual · Replays alternate paths to explore what would happen under different execution choices.
agenttrace/counterfactual/replay.py - Loop detect · Finds and classifies cycles, with embedding support for loop-related analysis.
agenttrace/loopdetect/cycle_finder.py - Classifier · Classifies detected loop patterns and supports semantic loop analysis.
agenttrace/loopdetect/classifier.py - Agent watchdog · Comparative analysis, semantic diffing, upstream divergence, and report generation for agenttrace.
agenttrace/watchdog/compare.py
Examples & fixtures
Demo agents, reports, and canonical run fixtures used to exercise the system.
- Example agent · Minimal LangGraph agent used to demonstrate recording and replay integration.
examples/simple_agent.py - Research demo · Larger demo agent and runner showing how the library is embedded in a real workflow.
demo/research_agent.py - Run fixtures · Representative clean, regressed, looped, retrying, and divergence runs used by tests and demos.
agenttrace/fixtures/clean_run_fixture.json
Connections 26
- Recorder → Storage: persist session · Recorded execution events are written to durable session storage.
- Storage → Replay: load run · Replay reconstructs executions from stored events.
- Storage → Diff: compare sessions · Diff reads multiple stored sessions to compute changes.
- Storage → Blame: analyze failure · Blame uses persisted state transitions and node boundaries.
- Storage → Diagnosis: explain failure · Diagnosis consumes stored sessions and blame results.
- Storage → Search: query corpus · Search indexes and retrieves historical sessions from storage.
- CLI → Recorder: record · CLI exposes the recording entrypoint around graph execution.
- CLI → Replay: replay · CLI launches replay of a stored session.
- CLI → Diff: diff · CLI compares sessions from the command line.
- CLI → Blame: blame · CLI surfaces failure attribution.
- CLI → Diagnosis: diagnose · CLI surfaces deeper root-cause analysis.
- CLI → Search: search · CLI provides natural-language or semantic lookup.
- CLI → Watchdog: watch · CLI starts regression monitoring and comparison.
- TUI → Storage: inspect · The TUI renders stored sessions and replay state.
- Watchdog → Replay: rerun · Watchdog replays saved sessions after source edits.
- Watchdog → Diff: regression check · Watchdog compares new behavior to prior baselines.
- Watchdog → Storage: baseline source · Watchdog relies on stored sessions as its comparison source.
- Annotations → Agent watchdog: label runs · Annotations feed labeled metadata into watchdog-style analysis.
- Counterfactual → Run fixtures: uses examples · Counterfactual replay is exercised by representative fixtures.
- Loop detect → Classifier: classify loops · Cycle detection feeds loop classification and semantic analysis.
- Agent watchdog → Storage: compare baselines · Agenttrace watchdog reads runs and baselines for divergence checks.
- Example agent → Recorder: emit events · Example workflows demonstrate the recording boundary.
- Research demo → Recorder: emit events · Demo workflows exercise recorder integration in a realistic agent.
- Run fixtures → Replay: validate replay · Canonical fixtures are used to verify reconstruction behavior.
- Run fixtures → Blame: validate blame · Fixtures encode expected failure and edge-case shapes.
- Run fixtures → Watchdog: validate regressions · Fixtures provide baseline and regressed runs for regression checks.