# peteromallet/dataclaw architecture

> Architecture diagram of the GitHub repository peteromallet/dataclaw, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peteromallet/dataclaw
- Repository: https://github.com/peteromallet/dataclaw

## Overview

1. **What the system is**
   DataClaw is a Python command-line tool, not a server app. Its job is to read local coding-agent conversation logs, normalize them into a single dataset schema, redact sensitive data, and optionally publish the result to a Hugging Face dataset repository.

2. **Primary runtime boundary**
   The whole system runs on the user’s machine and interacts with three external surfaces: local agent log directories, the user’s local configuration/state, and the Hugging Face Hub for auth and upload. There is no evidence of a long-running backend service.

3. **CLI orchestration layer**
   `dataclaw/cli.py` is the top-level entrypoint, with the actual command wiring split across `dataclaw/_cli/commands.py`, `common.py`, `exporting.py`, and `review.py`. The command flow is intentionally staged: prep → configure scope → export locally → review/confirm → publish.

4. **Source-specific parsing subsystem**
   The `dataclaw/parsers/` package is the main adapter layer. It contains provider-specific parsers for Claude, Codex, Cursor, Gemini, Kimi, OpenClaw, OpenCode, and a custom path. These parsers hide format differences in each agent’s raw logs and map them into one internal conversation model.

5. **Common normalization layer**
   Shared parsing logic lives in `dataclaw/parsers/common.py` and `dataclaw/parser.py`. This layer is where provider-specific events are turned into a unified session structure: session metadata, messages, tool calls, timestamps, model identifiers, branch info, and token statistics.

6. **Output schema and serialization**
   Exported data is written as JSONL, one session per line, plus a `metadata.json` summary for the Hugging Face repo. `dataclaw/_json.py` likely handles deterministic JSON serialization, while the README’s schema shows the canonical export contract: messages, tool uses, content parts, and aggregate stats.

7. **Privacy and redaction pipeline**
   The privacy boundary is enforced by `dataclaw/anonymizer.py` and `dataclaw/secrets.py`. This layer redacts usernames, configurable strings, emails, common API/secret patterns, and high-entropy suspicious strings. It also applies the same protections to tool-call inputs/outputs, which is important because tool traffic can contain leaked secrets even when chat text does not.

8. **Configuration and export scope control**
   `dataclaw/config.py` manages the user’s persistent choices: target HF repo, source scope (`claude`, `codex`, `all`, etc.), excluded projects, manual redaction strings, username redactions, and confirmation state. This is the mechanism that lets the CLI narrow what gets exported without changing the raw source logs.

9. **Review / attestation gate before publish**
   The `confirm` workflow is a separate safety gate, not just a formality. It requires the user to acknowledge review of PII findings, manual scanning, and either an exact-name scan or an explicit skip attestation. Architecturally, this gate prevents the publish step from being reachable until review criteria are satisfied.

10. **Publish path**
   The export command can run in local-only mode first, then push to Hugging Face only after approval. Hugging Face is the only notable external service in the runtime path, and its role is both authentication and dataset hosting. The README makes clear that the final artifact is a tagged HF dataset repo.

11. **Claude Code skill integration**
   The `.claude/skills/dataclaw/SKILL.md` asset and the `update-skill claude` command are an integration layer for Claude Code specifically. This is separate from the core exporter: it helps the agent follow the DataClaw workflow, but it does not change the export architecture itself.

12. **Validation and maintenance surface**
   The `tests/` directory mirrors the architecture almost exactly: anonymizer, config, CLI wrappers, and per-provider parsers all have dedicated tests. That tells you the core design is contract-based: provider adapters must produce the same normalized schema, and the privacy/export pipeline must stay stable across sources. GitHub Actions and pre-commit support reinforce that this is a packaged CLI project with CI validation rather than a deployed service.

Main entry points: [`dataclaw/parser.py`](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parser.py) (Parser API), [`tests`](https://github.com/peteromallet/dataclaw/tree/main/tests) (Tests), [`dataclaw/_cli/exporting.py`](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/exporting.py) (Export flow), [`dataclaw/_cli/commands.py`](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/commands.py) (Commands)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_cli_entry["CLI entry<br/>entrypoint<br/>[cli.py]"]
  node_cli_commands["Commands<br/>command router<br/>[commands.py]"]
  node_cli_common["CLI common<br/>cli helpers<br/>[common.py]"]
  node_cli_exporting["Export flow<br/>export orchestration<br/>[exporting.py]"]
  node_cli_review["Review gate<br/>attestation gate<br/>[review.py]"]
  node_config["Config<br/>persistent state<br/>[config.py]"]
end

subgraph group_core["Core pipeline"]
  node_providers["Providers<br/>source registry<br/>[providers.py]"]
  node_parser_facade["Parser API<br/>facade<br/>[parser.py]"]
  node_parser_common["Normalize<br/>schema mapper<br/>[common.py]"]
  node_parser_claude["Claude<br/>source parser<br/>[claude.py]"]
  node_parser_codex["Codex<br/>source parser<br/>[codex.py]"]
  node_parser_cursor["Cursor<br/>source parser<br/>[cursor.py]"]
  node_parser_gemini["Gemini<br/>source parser<br/>[gemini.py]"]
  node_parser_opencode["OpenCode<br/>source parser<br/>[opencode.py]"]
  node_anonymizer["Redaction<br/>privacy filter<br/>[anonymizer.py]"]
  node_secrets["Secret rules<br/>pattern rules<br/>[secrets.py]"]
  node_json_util["JSON util<br/>serializer<br/>[_json.py]"]
end

subgraph group_integrations["Integrations"]
  node_publish(("HF publish<br/>hub client"))
  node_skill["Claude skill<br/>agent integration<br/>[SKILL.md]"]
end

subgraph group_maintenance["Maintenance"]
  node_tests["Tests<br/>validation suite"]
end

node_cli_entry -->|"dispatches"| node_cli_commands
node_cli_commands -->|"uses"| node_cli_common
node_cli_commands -->|"orchestrates"| node_cli_exporting
node_cli_exporting -->|"gates"| node_cli_review
node_cli_common -->|"reads/writes"| node_config
node_cli_exporting -->|"invokes"| node_parser_facade
node_parser_facade -->|"selects"| node_providers
node_parser_facade -->|"normalizes via"| node_parser_common
node_parser_facade -->|"loads"| node_parser_claude
node_parser_facade -->|"loads"| node_parser_codex
node_parser_facade -->|"loads"| node_parser_cursor
node_parser_facade -->|"loads"| node_parser_gemini
node_parser_facade -->|"loads"| node_parser_opencode
node_parser_common -->|"passes through"| node_anonymizer
node_anonymizer -->|"matches"| node_secrets
node_anonymizer -->|"serializes"| node_json_util
node_cli_exporting -->|"pushes"| node_publish
node_config -->|"configures"| node_publish
node_skill -.->|"guides"| node_cli_entry
node_tests -.->|"covers"| node_cli_commands
node_tests -.->|"covers"| node_parser_common
node_tests -.->|"covers"| node_anonymizer
node_tests -.->|"covers"| node_config
node_tests -.->|"covers"| node_publish

click node_cli_entry "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/cli.py"
click node_cli_commands "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/commands.py"
click node_cli_common "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/common.py"
click node_cli_exporting "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/exporting.py"
click node_cli_review "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/review.py"
click node_config "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/config.py"
click node_providers "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/providers.py"
click node_parser_facade "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parser.py"
click node_parser_common "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/common.py"
click node_parser_claude "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/claude.py"
click node_parser_codex "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/codex.py"
click node_parser_cursor "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/cursor.py"
click node_parser_gemini "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/gemini.py"
click node_parser_opencode "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/opencode.py"
click node_anonymizer "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/anonymizer.py"
click node_secrets "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/secrets.py"
click node_json_util "https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_json.py"
click node_skill "https://github.com/peteromallet/dataclaw/blob/main/.claude/skills/dataclaw/SKILL.md"
click node_tests "https://github.com/peteromallet/dataclaw/tree/main/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_entry,node_cli_commands,node_cli_common,node_cli_exporting,node_cli_review,node_config toneBlue
class node_providers,node_parser_facade,node_parser_common,node_parser_claude,node_parser_codex,node_parser_cursor,node_parser_gemini,node_parser_opencode,node_anonymizer,node_secrets,node_json_util toneAmber
class node_publish,node_skill toneMint
class node_tests toneRose
```

## Components

### Runtime

Local CLI execution and external surfaces

- [CLI entry](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/cli.py): entrypoint · Top-level command entry and dispatch · `dataclaw/cli.py`
- [Commands](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/commands.py): command router · Wires subcommands and orchestration steps · `dataclaw/_cli/commands.py`
- [CLI common](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/common.py): cli helpers · Shared CLI arguments and scope setup · `dataclaw/_cli/common.py`
- [Export flow](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/exporting.py): export orchestration · Local export, review, and publish flow · `dataclaw/_cli/exporting.py`
- [Review gate](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_cli/review.py): attestation gate · Confirmation and PII review checks before publish · `dataclaw/_cli/review.py`
- [Config](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/config.py): persistent state · User scope, repo, exclusions, and redaction settings · `dataclaw/config.py`

### Core pipeline

Parsing, normalization, redaction, and serialization

- [Providers](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/providers.py): source registry · Maps supported coding-agent sources · `dataclaw/providers.py`
- [Parser API](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parser.py): facade · Selects and drives source parsers · `dataclaw/parser.py`
- [Normalize](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/common.py): schema mapper · Converts raw events into unified sessions · `dataclaw/parsers/common.py`
- [Claude](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/claude.py): source parser · Claude log adapter · `dataclaw/parsers/claude.py`
- [Codex](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/codex.py): source parser · Codex log adapter · `dataclaw/parsers/codex.py`
- [Cursor](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/cursor.py): source parser · Cursor log adapter · `dataclaw/parsers/cursor.py`
- [Gemini](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/gemini.py): source parser · Gemini log adapter · `dataclaw/parsers/gemini.py`
- [OpenCode](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/parsers/opencode.py): source parser · OpenCode log adapter · `dataclaw/parsers/opencode.py`
- [Redaction](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/anonymizer.py): privacy filter · Redacts names, secrets, and suspicious strings · `dataclaw/anonymizer.py`
- [Secret rules](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/secrets.py): pattern rules · Sensitive-pattern detection used by redaction · `dataclaw/secrets.py`
- [JSON util](https://github.com/peteromallet/dataclaw/blob/main/dataclaw/_json.py): serializer · Deterministic JSON serialization helpers · `dataclaw/_json.py`

### Integrations

Publishing and Claude Code support

- **HF publish**: hub client · Authenticates and uploads dataset artifacts
- [Claude skill](https://github.com/peteromallet/dataclaw/blob/main/.claude/skills/dataclaw/SKILL.md): agent integration · Claude Code workflow guidance asset · `.claude/skills/dataclaw/SKILL.md`

### Maintenance

Tests and CI support

- [Tests](https://github.com/peteromallet/dataclaw/tree/main/tests): validation suite · Architecture-level coverage for parsers, CLI, config, and privacy · `tests`

## Connections

- CLI entry → Commands: dispatches · CLI entrypoint hands off to command wiring
- Commands → CLI common: uses · Commands share argument and scope helpers
- Commands → Export flow: orchestrates · Command flow advances into export pipeline
- Export flow → Review gate: gates · Publish is blocked until review attestation passes
- CLI common → Config: reads/writes · CLI scope and user settings come from persistent config
- Export flow → Parser API: invokes · Export flow triggers parsing of local logs
- Parser API → Providers: selects · Facade chooses parser based on source provider
- Parser API → Normalize: normalizes via · Unified session schema is produced by common parser logic
- Parser API → Claude: loads · Claude-specific adapter
- Parser API → Codex: loads · Codex-specific adapter
- Parser API → Cursor: loads · Cursor-specific adapter
- Parser API → Gemini: loads · Gemini-specific adapter
- Parser API → OpenCode: loads · OpenCode-specific adapter
- Normalize → Redaction: passes through · Normalized text and tool traffic are redacted
- Redaction → Secret rules: matches · Secret-pattern detection powers redaction decisions
- Redaction → JSON util: serializes · Redacted sessions are prepared for JSONL output
- Export flow → HF publish: pushes · Approved exports are uploaded to Hugging Face
- Config → HF publish: configures · Target repo and publish options come from user config
- Claude skill → CLI entry: guides · Claude Code skill documents the workflow
- Tests → Commands: covers · CLI command behavior is validated
- Tests → Normalize: covers · Normalization contract is validated
- Tests → Redaction: covers · Privacy filtering is validated
- Tests → Config: covers · Persistent configuration behavior is validated
- Tests → HF publish: covers · Publish path is validated
