# dhanesh10war/chunklens architecture

> Architecture diagram of the GitHub repository dhanesh10war/chunklens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhanesh10war/chunklens
- Repository: https://github.com/dhanesh10war/chunklens

## Overview

ChunkLens is a browser-only workbench for experimenting with document chunking and estimating token and embedding costs. Users start with a sample or provide a document, configure chunking, then inspect the resulting chunks, compare configurations, or analyze similarity and tokenizer costs; exports turn the chosen results into code or datasets. The excerpts show App coordinating ingestion, worker execution, and views, and show the worker dispatching to chunking logic. Several capabilities are documented but their internal wiring was not sampled, so those links are omitted or marked unverified.

Main entry points: [`src/App.tsx`](https://github.com/dhanesh10war/chunklens/blob/main/src/App.tsx) (Workbench app), [`src/engine/worker/useChunkingWorker.ts`](https://github.com/dhanesh10war/chunklens/blob/main/src/engine/worker/useChunkingWorker.ts) (Chunking worker hook), [`src/components/ExportModal.tsx`](https://github.com/dhanesh10war/chunklens/blob/main/src/components/ExportModal.tsx) (Export interface), [`src/hooks/useDocumentUpload.ts`](https://github.com/dhanesh10war/chunklens/blob/main/src/hooks/useDocumentUpload.ts) (Document upload)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Workbench workflow"]
  node_app["Workbench app<br/>[App.tsx]"]
  node_samples["Sample documents<br/>[samples.ts]"]
  node_sidebar["Strategy controls<br/>[Sidebar.tsx]"]
end

subgraph group_processing["Document processing"]
  node_upload["Document upload"]
  node_format["Format detection"]
  node_pdf["PDF parsing<br/>[pdfParser.ts]"]
  node_workerhook["Chunking worker hook"]
  node_worker["Chunking worker<br/>[chunking.worker.ts]"]
  node_chunkers["Chunking strategies"]
end

subgraph group_analysis["Analysis and inspection"]
  node_canvas["Chunk canvas<br/>[VisualCanvas.tsx]"]
  node_inspector["Chunk inspector<br/>[ChunkInspector.tsx]"]
  node_comparison["A/B comparison<br/>[ComparisonView.tsx]"]
  node_similarity["Similarity heatmap"]
  node_vectorizer["TF-IDF vectorizer<br/>[vectorizer.ts]"]
  node_metrics["Chunk metrics<br/>[MetricsBar.tsx]"]
  node_tokenmodal["Tokenizer and cost analysis"]
  node_multimodel["Multi-model tokenizer<br/>[multiModel.ts]"]
end

subgraph group_output["Export and session"]
  node_exportmodal["Export interface<br/>[ExportModal.tsx]"]
  node_codeexport["Code exporters<br/>[codeExporters.ts]"]
  node_dataexport["Dataset exporters<br/>[dataExporters.ts]"]
  node_session["Session storage<br/>[storage.ts]"]
end

node_engineer(("Engineer"))

node_engineer -->|"uses"| node_app
node_app -->|"selects"| node_samples
node_app -->|"handles upload"| node_upload
node_app -->|"detects format"| node_format
node_app -->|"requests chunking"| node_workerhook
node_workerhook -->|"dispatches work"| node_worker
node_workerhook -->|"runs fallback"| node_chunkers
node_worker -->|"executes chunking"| node_chunkers
node_app -->|"renders controls"| node_sidebar
node_app -->|"renders chunks"| node_canvas
node_app -->|"renders details"| node_inspector
node_app -->|"renders comparison"| node_comparison
node_app -->|"renders analysis"| node_similarity
node_similarity -.->|"computes similarity"| node_vectorizer
node_app -->|"renders metrics"| node_metrics
node_app -->|"opens analysis"| node_tokenmodal
node_tokenmodal -->|"compares tokenizers"| node_multimodel
node_app -->|"opens export"| node_exportmodal
node_exportmodal -.->|"generates code"| node_codeexport
node_exportmodal -.->|"exports datasets"| node_dataexport
node_app -->|"saves session"| node_session
node_upload -.->|"parses PDFs"| node_pdf

click node_app "https://github.com/dhanesh10war/chunklens/blob/main/src/App.tsx"
click node_samples "https://github.com/dhanesh10war/chunklens/blob/main/src/data/samples.ts"
click node_upload "https://github.com/dhanesh10war/chunklens/blob/main/src/hooks/useDocumentUpload.ts"
click node_format "https://github.com/dhanesh10war/chunklens/blob/main/src/utils/fileFormatDetector.ts"
click node_pdf "https://github.com/dhanesh10war/chunklens/blob/main/src/utils/pdfParser.ts"
click node_workerhook "https://github.com/dhanesh10war/chunklens/blob/main/src/engine/worker/useChunkingWorker.ts"
click node_worker "https://github.com/dhanesh10war/chunklens/blob/main/src/engine/worker/chunking.worker.ts"
click node_chunkers "https://github.com/dhanesh10war/chunklens/tree/main/src/engine/chunkers"
click node_sidebar "https://github.com/dhanesh10war/chunklens/blob/main/src/components/Sidebar.tsx"
click node_canvas "https://github.com/dhanesh10war/chunklens/blob/main/src/components/VisualCanvas.tsx"
click node_inspector "https://github.com/dhanesh10war/chunklens/blob/main/src/components/ChunkInspector.tsx"
click node_comparison "https://github.com/dhanesh10war/chunklens/blob/main/src/components/ComparisonView.tsx"
click node_similarity "https://github.com/dhanesh10war/chunklens/blob/main/src/components/SimilarityHeatmap.tsx"
click node_vectorizer "https://github.com/dhanesh10war/chunklens/blob/main/src/engine/embeddings/vectorizer.ts"
click node_metrics "https://github.com/dhanesh10war/chunklens/blob/main/src/components/MetricsBar.tsx"
click node_tokenmodal "https://github.com/dhanesh10war/chunklens/blob/main/src/components/TokenizerComparisonModal.tsx"
click node_multimodel "https://github.com/dhanesh10war/chunklens/blob/main/src/engine/tokenizers/multiModel.ts"
click node_exportmodal "https://github.com/dhanesh10war/chunklens/blob/main/src/components/ExportModal.tsx"
click node_codeexport "https://github.com/dhanesh10war/chunklens/blob/main/src/exporters/codeExporters.ts"
click node_dataexport "https://github.com/dhanesh10war/chunklens/blob/main/src/exporters/dataExporters.ts"
click node_session "https://github.com/dhanesh10war/chunklens/blob/main/src/utils/storage.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_samples,node_sidebar toneBlue
class node_upload,node_format,node_pdf,node_workerhook,node_worker,node_chunkers toneAmber
class node_canvas,node_inspector,node_comparison,node_similarity,node_vectorizer,node_metrics,node_tokenmodal,node_multimodel toneMint
class node_exportmodal,node_codeexport,node_dataexport,node_session toneRose
class node_engineer toneIndigo
```

## Components

### Workbench workflow

- [Workbench app](https://github.com/dhanesh10war/chunklens/blob/main/src/App.tsx): `src/App.tsx`
- [Sample documents](https://github.com/dhanesh10war/chunklens/blob/main/src/data/samples.ts): `src/data/samples.ts`
- [Strategy controls](https://github.com/dhanesh10war/chunklens/blob/main/src/components/Sidebar.tsx): `src/components/Sidebar.tsx`

### Document processing

- [Document upload](https://github.com/dhanesh10war/chunklens/blob/main/src/hooks/useDocumentUpload.ts): `src/hooks/useDocumentUpload.ts`
- [Format detection](https://github.com/dhanesh10war/chunklens/blob/main/src/utils/fileFormatDetector.ts): `src/utils/fileFormatDetector.ts`
- [PDF parsing](https://github.com/dhanesh10war/chunklens/blob/main/src/utils/pdfParser.ts): `src/utils/pdfParser.ts`
- [Chunking worker hook](https://github.com/dhanesh10war/chunklens/blob/main/src/engine/worker/useChunkingWorker.ts): `src/engine/worker/useChunkingWorker.ts`
- [Chunking worker](https://github.com/dhanesh10war/chunklens/blob/main/src/engine/worker/chunking.worker.ts): `src/engine/worker/chunking.worker.ts`
- [Chunking strategies](https://github.com/dhanesh10war/chunklens/tree/main/src/engine/chunkers): `src/engine/chunkers`

### Analysis and inspection

- [Chunk canvas](https://github.com/dhanesh10war/chunklens/blob/main/src/components/VisualCanvas.tsx): `src/components/VisualCanvas.tsx`
- [Chunk inspector](https://github.com/dhanesh10war/chunklens/blob/main/src/components/ChunkInspector.tsx): `src/components/ChunkInspector.tsx`
- [A/B comparison](https://github.com/dhanesh10war/chunklens/blob/main/src/components/ComparisonView.tsx): `src/components/ComparisonView.tsx`
- [Similarity heatmap](https://github.com/dhanesh10war/chunklens/blob/main/src/components/SimilarityHeatmap.tsx): `src/components/SimilarityHeatmap.tsx`
- [TF-IDF vectorizer](https://github.com/dhanesh10war/chunklens/blob/main/src/engine/embeddings/vectorizer.ts): `src/engine/embeddings/vectorizer.ts`
- [Chunk metrics](https://github.com/dhanesh10war/chunklens/blob/main/src/components/MetricsBar.tsx): `src/components/MetricsBar.tsx`
- [Tokenizer and cost analysis](https://github.com/dhanesh10war/chunklens/blob/main/src/components/TokenizerComparisonModal.tsx): `src/components/TokenizerComparisonModal.tsx`
- [Multi-model tokenizer](https://github.com/dhanesh10war/chunklens/blob/main/src/engine/tokenizers/multiModel.ts): `src/engine/tokenizers/multiModel.ts`

### Export and session

- [Export interface](https://github.com/dhanesh10war/chunklens/blob/main/src/components/ExportModal.tsx): `src/components/ExportModal.tsx`
- [Code exporters](https://github.com/dhanesh10war/chunklens/blob/main/src/exporters/codeExporters.ts): `src/exporters/codeExporters.ts`
- [Dataset exporters](https://github.com/dhanesh10war/chunklens/blob/main/src/exporters/dataExporters.ts): `src/exporters/dataExporters.ts`
- [Session storage](https://github.com/dhanesh10war/chunklens/blob/main/src/utils/storage.ts): `src/utils/storage.ts`

### Other components

- **Engineer**

## Connections

- Engineer → Workbench app: uses (evidence: `README.md`)
- Workbench app → Sample documents: selects (evidence: `src/App.tsx`)
- Workbench app → Document upload: handles upload (evidence: `src/App.tsx`)
- Workbench app → Format detection: detects format (evidence: `src/App.tsx`)
- Workbench app → Chunking worker hook: requests chunking (evidence: `src/App.tsx`)
- Chunking worker hook → Chunking worker: dispatches work (evidence: `src/engine/worker/useChunkingWorker.ts`)
- Chunking worker hook → Chunking strategies: runs fallback (evidence: `src/engine/worker/useChunkingWorker.ts`)
- Chunking worker → Chunking strategies: executes chunking (evidence: `src/engine/worker/chunking.worker.ts`)
- Workbench app → Strategy controls: renders controls (evidence: `src/App.tsx`)
- Workbench app → Chunk canvas: renders chunks (evidence: `src/App.tsx`)
- Workbench app → Chunk inspector: renders details (evidence: `src/App.tsx`)
- Workbench app → A/B comparison: renders comparison (evidence: `src/App.tsx`)
- Workbench app → Similarity heatmap: renders analysis (evidence: `src/App.tsx`)
- Similarity heatmap → TF-IDF vectorizer: computes similarity
- Workbench app → Chunk metrics: renders metrics (evidence: `src/App.tsx`)
- Workbench app → Tokenizer and cost analysis: opens analysis (evidence: `src/App.tsx`)
- Tokenizer and cost analysis → Multi-model tokenizer: compares tokenizers (evidence: `src/components/TokenizerComparisonModal.tsx`)
- Workbench app → Export interface: opens export (evidence: `src/App.tsx`)
- Export interface → Code exporters: generates code
- Export interface → Dataset exporters: exports datasets
- Workbench app → Session storage: saves session (evidence: `src/App.tsx`)
- Document upload → PDF parsing: parses PDFs (evidence: `README.md`)
