How peteromallet/dataclaw works
1. What the system is
DataClaw is a Python command-line tool, not a server app. Its job is to read local coding-agent conversation logs, normalize them into a single dataset schema, redact sensitive data, and optionally publish the result to a Hugging Face dataset repository.
2. Primary runtime boundary
The whole system runs on the user’s machine and interacts with three external surfaces: local agent log directories, the user’s local configuration/state, and the Hugging Face Hub for auth and upload. There is no evidence of a long-running backend service.
3. CLI orchestration layer
dataclaw/cli.py is the top-level entrypoint, with the actual command wiring split across dataclaw/_cli/commands.py, common.py, exporting.py, and review.py. The command flow is intentionally staged: prep → configure scope → export locally → review/confirm → publish.
4. Source-specific parsing subsystem
The dataclaw/parsers/ package is the main adapter layer. It contains provider-specific parsers for Claude, Codex, Cursor, Gemini, Kimi, OpenClaw, OpenCode, and a custom path. These parsers hide format differences in each agent’s raw logs and map them into one internal conversation model.
5. Common normalization layer
Shared parsing logic lives in dataclaw/parsers/common.py and dataclaw/parser.py. This layer is where provider-specific events are turned into a unified session structure: session metadata, messages, tool calls, timestamps, model identifiers, branch info, and token statistics.
6. Output schema and serialization
Exported data is written as JSONL, one session per line, plus a metadata.json summary for the Hugging Face repo. dataclaw/_json.py likely handles deterministic JSON serialization, while the README’s schema shows the canonical export contract: messages, tool uses, content parts, and aggregate stats.
7. Privacy and redaction pipeline
The privacy boundary is enforced by dataclaw/anonymizer.py and dataclaw/secrets.py. This layer redacts usernames, configurable strings, emails, common API/secret patterns, and high-entropy suspicious strings. It also applies the same protections to tool-call inputs/outputs, which is important because tool traffic can contain leaked secrets even when chat text does not.
8. Configuration and export scope control
dataclaw/config.py manages the user’s persistent choices: target HF repo, source scope (claude, codex, all, etc.), excluded projects, manual redaction strings, username redactions, and confirmation state. This is the mechanism that lets the CLI narrow what gets exported without changing the raw source logs.
9. Review / attestation gate before publish
The confirm workflow is a separate safety gate, not just a formality. It requires the user to acknowledge review of PII findings, manual scanning, and either an exact-name scan or an explicit skip attestation. Architecturally, this gate prevents the publish step from being reachable until review criteria are satisfied.
10. Publish path
The export command can run in local-only mode first, then push to Hugging Face only after approval. Hugging Face is the only notable external service in the runtime path, and its role is both authentication and dataset hosting. The README makes clear that the final artifact is a tagged HF dataset repo.
11. Claude Code skill integration
The .claude/skills/dataclaw/SKILL.md asset and the update-skill claude command are an integration layer for Claude Code specifically. This is separate from the core exporter: it helps the agent follow the DataClaw workflow, but it does not change the export architecture itself.
12. Validation and maintenance surface
The tests/ directory mirrors the architecture almost exactly: anonymizer, config, CLI wrappers, and per-provider parsers all have dedicated tests. That tells you the core design is contract-based: provider adapters must produce the same normalized schema, and the privacy/export pipeline must stay stable across sources. GitHub Actions and pre-commit support reinforce that this is a packaged CLI project with CI validation rather than a deployed service.
Main entry points: parser.py Parser API · tests Tests · exporting.py Export flow · commands.py Commands
Components 20
Runtime
Local CLI execution and external surfaces
- CLI entry · Top-level command entry and dispatch
dataclaw/cli.py - Commands · Wires subcommands and orchestration steps
dataclaw/_cli/commands.py - CLI common · Shared CLI arguments and scope setup
dataclaw/_cli/common.py - Export flow · Local export, review, and publish flow
dataclaw/_cli/exporting.py - Review gate · Confirmation and PII review checks before publish
dataclaw/_cli/review.py - Config · User scope, repo, exclusions, and redaction settings
dataclaw/config.py
Core pipeline
Parsing, normalization, redaction, and serialization
- Providers · Maps supported coding-agent sources
dataclaw/providers.py - Parser API · Selects and drives source parsers
dataclaw/parser.py - Normalize · Converts raw events into unified sessions
dataclaw/parsers/common.py - Claude · Claude log adapter
dataclaw/parsers/claude.py - Codex · Codex log adapter
dataclaw/parsers/codex.py - Cursor · Cursor log adapter
dataclaw/parsers/cursor.py - Gemini · Gemini log adapter
dataclaw/parsers/gemini.py - OpenCode · OpenCode log adapter
dataclaw/parsers/opencode.py - Redaction · Redacts names, secrets, and suspicious strings
dataclaw/anonymizer.py - Secret rules · Sensitive-pattern detection used by redaction
dataclaw/secrets.py - JSON util · Deterministic JSON serialization helpers
dataclaw/_json.py
Integrations
Publishing and Claude Code support
- HF publish · Authenticates and uploads dataset artifacts
- Claude skill · Claude Code workflow guidance asset
.claude/skills/dataclaw/SKILL.md
Maintenance
Tests and CI support
- Tests · Architecture-level coverage for parsers, CLI, config, and privacy
tests
Connections 24
- CLI entry → Commands: dispatches · CLI entrypoint hands off to command wiring
- Commands → CLI common: uses · Commands share argument and scope helpers
- Commands → Export flow: orchestrates · Command flow advances into export pipeline
- Export flow → Review gate: gates · Publish is blocked until review attestation passes
- CLI common → Config: reads/writes · CLI scope and user settings come from persistent config
- Export flow → Parser API: invokes · Export flow triggers parsing of local logs
- Parser API → Providers: selects · Facade chooses parser based on source provider
- Parser API → Normalize: normalizes via · Unified session schema is produced by common parser logic
- Parser API → Claude: loads · Claude-specific adapter
- Parser API → Codex: loads · Codex-specific adapter
- Parser API → Cursor: loads · Cursor-specific adapter
- Parser API → Gemini: loads · Gemini-specific adapter
- Parser API → OpenCode: loads · OpenCode-specific adapter
- Normalize → Redaction: passes through · Normalized text and tool traffic are redacted
- Redaction → Secret rules: matches · Secret-pattern detection powers redaction decisions
- Redaction → JSON util: serializes · Redacted sessions are prepared for JSONL output
- Export flow → HF publish: pushes · Approved exports are uploaded to Hugging Face
- Config → HF publish: configures · Target repo and publish options come from user config
- Claude skill → CLI entry: guides · Claude Code skill documents the workflow
- Tests → Commands: covers · CLI command behavior is validated
- Tests → Normalize: covers · Normalization contract is validated
- Tests → Redaction: covers · Privacy filtering is validated
- Tests → Config: covers · Persistent configuration behavior is validated
- Tests → HF publish: covers · Publish path is validated