How zerosumquant/claude-conversation-extractor works
1. What this repo is
- A local, offline Python tool that reads Claude Code conversation logs from the user’s machine and turns them into searchable/exportable artifacts.
- There is no backend, no network API, and no database; the primary boundary is the local filesystem under ~/.claude/projects.
2. Primary data source
- The input is Claude Code’s undocumented chat_*.jsonl files stored in per-project directories.
- The tool scans that directory tree, discovers conversation sessions, and treats those files as the source of truth for both export and search.
3. Core extraction pipeline
- src/extract_claude_logs.py appears to be the central extraction module.
- Its role is to locate logs, parse JSONL records, normalize conversation structure, and emit exports in human-readable formats such as Markdown, plus structured formats like JSON and HTML.
- This module is the main “read filesystem → parse messages → render output” path.
4. Search subsystem
- src/search_conversations.py provides search over the extracted conversation corpus.
- The search flow is local and text-based: scan conversations, match query terms, return hits with context/previews, and optionally feed matches into extraction/export actions.
- README also suggests case-insensitive full-text search and direct search from the CLI.
5. Real-time search subsystem
- src/realtime_search.py is a distinct interactive search engine path that updates results as the user types.
- The dedicated tests for threading and UI imply this module coordinates incremental filtering, event handling, and background work so the interface stays responsive while searching locally large log sets.
- If optional semantic search is enabled, it would be an extension of this layer rather than a separate service.
6. Interactive UI boundary
- src/interactive_ui.py is the menu-driven front end used by claude-start.
- It wraps discovery, search, and export operations into a user-facing terminal interface with navigation, prompts, and real-time search integration.
- The UI is orchestration-only; it does not own storage or parsing logic.
7. CLI boundary
- src/search_cli.py and the README’s claude-extract, claude-search, and backward-compatible claude-logs commands indicate multiple command entry points over the same core library.
- The CLI layer is thin and delegates to the core extraction/search modules, making it suitable for scripting and automation.
- The architecture is therefore “shared engine, multiple front doors” rather than separate implementations.
8. Output and export formats
- The repo supports at least Markdown by default, plus JSON and HTML exports.
- Output is generated from the normalized conversation model, not directly from raw JSONL, so formatting concerns are separated from parsing concerns.
- The “detailed” export mode broadens the rendered transcript to include tool calls, MCP responses, system messages, and other metadata.
9. Runtime and dependency model
- The project is pure Python and cross-platform, targeting Python 3.8+.
- Core functionality is designed to work without external services; optional capabilities may add dependencies such as spaCy, but those are additive rather than required for the main flow.
- Packaging is handled via setup.py and pyproject.toml, which likely expose the command-line entry points used in the README.
10. Validation and quality gates
- The tests/ tree shows strong coverage of extraction, search, UI, threading, integration, and error handling.
- This mirrors the architecture: each major subsystem has dedicated tests, and there are integration tests that validate the seams between filesystem scanning, parsing, search, and UI behavior.
- CI workflows in .github/workflows/ run tests and publishing, so the release boundary is standard Python packaging rather than a deployed service.
11. Documentation as an architectural aid
- docs/user and docs/technical document usage, install, changelog, and SEO/technical notes, but they do not add runtime components.
- Their main architectural value is clarifying the intended user flows: discover logs, search them, then export selected sessions or all sessions.
12. How the main flow fits together
- The practical data flow is: local Claude JSONL files → discovery/scanning → parsing/normalization → search index or filtered result set → rendering into Markdown/JSON/HTML → file output.
- The interactive UI and CLI both sit on top of that same pipeline, so the architecture is best drawn as one core local processing engine with two presentation layers.
Main entry points: extract_claude_logs.py Extractor · interactive_ui.py Terminal UI · search_conversations.py Search core · test.yml CI workflows
Components 18
Core engine
Local scanning, parsing, search, and rendering pipeline over Claude conversation logs.
- Claude logs · Undocumented chat_*.jsonl conversation files under ~/.claude/projects.
- Discovery · Finds project directories and locates conversation sessions on disk.
src/extract_claude_logs.py - Parse JSONL · Reads raw log records and normalizes them into conversation structure.
src/extract_claude_logs.py - Render exports · Formats normalized conversations into Markdown, JSON, HTML, and detailed transcript views.
src/extract_claude_logs.py - Extractor · Main read-parse-normalize-export pipeline for local conversation logs.
src/extract_claude_logs.py - Search core · Searches extracted conversations with local text matching and context previews.
src/search_conversations.py - Realtime engine · Incremental search loop that updates results as the user types.
src/realtime_search.py
Interfaces
Command-line and terminal UI entry points that orchestrate the shared core.
- Search CLI · Thin command-line entry point for scripted search and extraction actions.
src/search_cli.py - Terminal UI · Menu-driven terminal interface that orchestrates discovery, search, and export.
src/interactive_ui.py
Quality and release
Tests, CI, and packaging that validate and ship the tool.
- Packaging · Project metadata and console entry points for installation and distribution.
setup.py - Build config · Project build and dependency configuration.
pyproject.toml - Dependencies · Base, dev, and optional dependency sets used across the tool.
requirements/base.txt - Extractor tests · Coverage for extraction, alignment, comprehensive parsing, and error handling.
tests/test_extractor.py - Search tests · Coverage for search behavior and search/extraction integration seams.
tests/test_search.py - Realtime tests · Coverage for threading, UI integration, and incremental search behavior.
tests/test_realtime_search_unit.py - UI tests · Coverage for the interactive terminal interface.
tests/test_interactive_ui.py - CI workflows · Test and publish workflows in GitHub Actions.
.github/workflows/test.yml
Docs
User and technical documentation that describe workflows and usage.
- User docs · Install, changelog, and technical/user-facing guidance.
docs/user/INSTALL.md
Connections 25
- Claude logs → Discovery: scan · Filesystem logs are discovered from the local Claude projects tree.
- Discovery → Parse JSONL: load · Discovered JSONL sessions are parsed into conversation records.
- Parse JSONL → Render exports: normalize · Parsed conversations are prepared for export rendering.
- Parse JSONL → Search core: index/search · Normalized conversations become the searchable corpus.
- Render exports → Extractor: serve · Rendering is part of the main extraction pipeline.
- Extractor → Search core: share corpus · The extractor and search path operate on the same conversation model.
- Search core → Realtime engine: interactive mode · Realtime search builds on the same local search logic.
- Search CLI → Search core: delegate · CLI commands call into shared search logic.
- Search CLI → Extractor: delegate · CLI commands can trigger extraction and export workflows.
- Terminal UI → Realtime engine: embed · The interactive UI incorporates realtime search for responsive filtering.
- Terminal UI → Extractor: orchestrate · The UI drives discovery and export actions without owning parsing logic.
- Packaging → Search CLI: expose · Packaging publishes command-line entry points.
- Packaging → Terminal UI: expose · Packaging exposes the terminal launcher.
- Build config → Packaging: configure · Build metadata and packaging settings support installation.
- Dependencies → Extractor: depends on · Core extraction relies on runtime dependencies declared in manifests.
- Extractor tests → Extractor: validate · Extraction tests exercise parsing and export behavior.
- Search tests → Search core: validate · Search tests verify matching, filtering, and integration behavior.
- Realtime tests → Realtime engine: validate · Realtime tests cover threading and incremental UI/search behavior.
- UI tests → Terminal UI: validate · UI tests verify menu-driven orchestration and prompts.
- CI workflows → Extractor tests: run · CI runs the extraction-focused test suite.
- CI workflows → Search tests: run · CI runs search validation.
- CI workflows → Realtime tests: run · CI runs realtime search validation.
- CI workflows → UI tests: run · CI runs interactive UI checks.
- User docs → Terminal UI: describe · Docs explain the interactive user flow.
- User docs → Search CLI: describe · Docs document command-line usage and workflows.