# 0xsero/ai-data-extraction architecture

> Architecture diagram of the GitHub repository 0xsero/ai-data-extraction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/0xsero/ai-data-extraction
- Repository: https://github.com/0xsero/ai-data-extraction

## Overview

This toolkit discovers local histories from multiple AI coding assistants, converts their vendor-specific records into conversation JSONL, and offers two downstream corpus workflows: privacy redaction and Agent Skills generation. The README documents the extractor coverage and output format; sampled source confirms several adapter parsing patterns and the downstream transformations. Some adapter internals and extract_all.sh were not sampled, so their finer-grained wiring is omitted.

Main entry points: [`extract_all.sh`](https://github.com/0xsero/ai-data-extraction/blob/main/extract_all.sh) (Extract all), [`filter_privacy.py`](https://github.com/0xsero/ai-data-extraction/blob/main/filter_privacy.py) (Privacy redactor), [`corpus_to_skills.py`](https://github.com/0xsero/ai-data-extraction/blob/main/corpus_to_skills.py) (Skill generator), [`extract_claude_code.py`](https://github.com/0xsero/ai-data-extraction/blob/main/extract_claude_code.py) (Claude extractor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_extractors["History extractors"]
  node_all["Extract all<br/>[extract_all.sh]"]
  node_claude["Claude extractor"]
  node_codex["Codex extractor<br/>[extract_codex.py]"]
  node_cursor["Cursor GUI extractor<br/>[extract_cursor.py]"]
  node_cursorcli["Cursor CLI extractor"]
  node_continue["Continue extractor"]
  node_gemini["Gemini extractor<br/>[extract_gemini.py]"]
  node_opencode["OpenCode extractor"]
  node_trae["Trae extractor<br/>[extract_trae.py]"]
  node_windsurf["Windsurf extractor"]
end

subgraph group_corpus["Corpus workflows"]
  node_jsonl["Conversation JSONL"]
  node_privacy["Privacy redactor<br/>[filter_privacy.py]"]
  node_redacted["Redacted JSONL"]
  node_skills["Skill generator"]
  node_skillsout["Generated skills"]
end

subgraph group_integrations["External integrations"]
  node_skillapi{{"Chat-completions API"}}
  node_opf{{"OpenAI Privacy Filter"}}
end

node_user(("Operator"))
node_localdata[("Assistant histories")]

node_user -->|"runs"| node_all
node_user -->|"runs"| node_claude
node_user -->|"runs"| node_codex
node_user -->|"runs"| node_cursor
node_user -->|"runs"| node_cursorcli
node_user -->|"runs"| node_continue
node_user -->|"runs"| node_gemini
node_user -->|"runs"| node_opencode
node_user -->|"runs"| node_trae
node_user -->|"runs"| node_windsurf
node_all -->|"dispatches"| node_claude
node_all -->|"dispatches"| node_codex
node_all -->|"dispatches"| node_cursor
node_all -->|"dispatches"| node_cursorcli
node_all -->|"dispatches"| node_continue
node_all -->|"dispatches"| node_gemini
node_all -->|"dispatches"| node_opencode
node_all -->|"dispatches"| node_trae
node_all -->|"dispatches"| node_windsurf
node_localdata -->|"reads"| node_claude
node_localdata -->|"reads"| node_codex
node_localdata -->|"reads"| node_cursor
node_localdata -->|"reads"| node_cursorcli
node_localdata -->|"reads"| node_continue
node_localdata -->|"reads"| node_gemini
node_localdata -->|"reads"| node_opencode
node_localdata -->|"reads"| node_trae
node_localdata -->|"reads"| node_windsurf
node_claude -->|"writes"| node_jsonl
node_codex -->|"writes"| node_jsonl
node_cursor -->|"writes"| node_jsonl
node_cursorcli -->|"writes"| node_jsonl
node_continue -->|"writes"| node_jsonl
node_gemini -->|"writes"| node_jsonl
node_opencode -->|"writes"| node_jsonl
node_trae -->|"writes"| node_jsonl
node_windsurf -->|"writes"| node_jsonl
node_user -->|"runs"| node_privacy
node_jsonl -->|"reads"| node_privacy
node_privacy -->|"loads"| node_opf
node_privacy -->|"writes"| node_redacted
node_user -->|"runs"| node_skills
node_jsonl -->|"reads"| node_skills
node_skills -->|"requests skills"| node_skillapi
node_skills -->|"writes"| node_skillsout

click node_all "https://github.com/0xsero/ai-data-extraction/blob/main/extract_all.sh"
click node_claude "https://github.com/0xsero/ai-data-extraction/blob/main/extract_claude_code.py"
click node_codex "https://github.com/0xsero/ai-data-extraction/blob/main/extract_codex.py"
click node_cursor "https://github.com/0xsero/ai-data-extraction/blob/main/extract_cursor.py"
click node_cursorcli "https://github.com/0xsero/ai-data-extraction/blob/main/extract_cursor_cli.py"
click node_continue "https://github.com/0xsero/ai-data-extraction/blob/main/extract_continue.py"
click node_gemini "https://github.com/0xsero/ai-data-extraction/blob/main/extract_gemini.py"
click node_opencode "https://github.com/0xsero/ai-data-extraction/blob/main/extract_opencode.py"
click node_trae "https://github.com/0xsero/ai-data-extraction/blob/main/extract_trae.py"
click node_windsurf "https://github.com/0xsero/ai-data-extraction/blob/main/extract_windsurf.py"
click node_privacy "https://github.com/0xsero/ai-data-extraction/blob/main/filter_privacy.py"
click node_skills "https://github.com/0xsero/ai-data-extraction/blob/main/corpus_to_skills.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_all,node_claude,node_codex,node_cursor,node_cursorcli,node_continue,node_gemini,node_opencode,node_trae,node_windsurf toneBlue
class node_jsonl,node_privacy,node_redacted,node_skills,node_skillsout,node_localdata toneAmber
class node_skillapi,node_opf toneMint
class node_user toneIndigo
```

## Components

### History extractors

- [Extract all](https://github.com/0xsero/ai-data-extraction/blob/main/extract_all.sh): `extract_all.sh`
- [Claude extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_claude_code.py): `extract_claude_code.py`
- [Codex extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_codex.py): `extract_codex.py`
- [Cursor GUI extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_cursor.py): `extract_cursor.py`
- [Cursor CLI extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_cursor_cli.py): `extract_cursor_cli.py`
- [Continue extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_continue.py): `extract_continue.py`
- [Gemini extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_gemini.py): `extract_gemini.py`
- [OpenCode extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_opencode.py): `extract_opencode.py`
- [Trae extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_trae.py): `extract_trae.py`
- [Windsurf extractor](https://github.com/0xsero/ai-data-extraction/blob/main/extract_windsurf.py): `extract_windsurf.py`

### Corpus workflows

- **Conversation JSONL**
- [Privacy redactor](https://github.com/0xsero/ai-data-extraction/blob/main/filter_privacy.py): `filter_privacy.py`
- **Redacted JSONL**
- [Skill generator](https://github.com/0xsero/ai-data-extraction/blob/main/corpus_to_skills.py): `corpus_to_skills.py`
- **Generated skills**

### External integrations

- **Chat-completions API**
- **OpenAI Privacy Filter**

### Other components

- **Operator**
- **Assistant histories**

## Connections

- Operator → Extract all: runs
- Operator → Claude extractor: runs
- Operator → Codex extractor: runs
- Operator → Cursor GUI extractor: runs
- Operator → Cursor CLI extractor: runs
- Operator → Continue extractor: runs
- Operator → Gemini extractor: runs
- Operator → OpenCode extractor: runs
- Operator → Trae extractor: runs
- Operator → Windsurf extractor: runs
- Extract all → Claude extractor: dispatches
- Extract all → Codex extractor: dispatches
- Extract all → Cursor GUI extractor: dispatches
- Extract all → Cursor CLI extractor: dispatches
- Extract all → Continue extractor: dispatches
- Extract all → Gemini extractor: dispatches
- Extract all → OpenCode extractor: dispatches
- Extract all → Trae extractor: dispatches
- Extract all → Windsurf extractor: dispatches
- Assistant histories → Claude extractor: reads
- Assistant histories → Codex extractor: reads
- Assistant histories → Cursor GUI extractor: reads
- Assistant histories → Cursor CLI extractor: reads
- Assistant histories → Continue extractor: reads
- Assistant histories → Gemini extractor: reads
- Assistant histories → OpenCode extractor: reads
- Assistant histories → Trae extractor: reads
- Assistant histories → Windsurf extractor: reads
- Claude extractor → Conversation JSONL: writes
- Codex extractor → Conversation JSONL: writes
- Cursor GUI extractor → Conversation JSONL: writes
- Cursor CLI extractor → Conversation JSONL: writes
- Continue extractor → Conversation JSONL: writes
- Gemini extractor → Conversation JSONL: writes
- OpenCode extractor → Conversation JSONL: writes
- Trae extractor → Conversation JSONL: writes
- Windsurf extractor → Conversation JSONL: writes
- Operator → Privacy redactor: runs
- Conversation JSONL → Privacy redactor: reads
- Privacy redactor → OpenAI Privacy Filter: loads
- Privacy redactor → Redacted JSONL: writes
- Operator → Skill generator: runs
- Conversation JSONL → Skill generator: reads
- Skill generator → Chat-completions API: requests skills
- Skill generator → Generated skills: writes