# arifshaon/webcrawlmanager architecture

> Architecture diagram of the GitHub repository arifshaon/webcrawlmanager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arifshaon/webcrawlmanager
- Repository: https://github.com/arifshaon/webcrawlmanager

## Overview

SWM is a local web-archive manager: a curator creates a job through the dashboard or CLI, a worker runs a browser-based capture, and SWM preserves the resulting WARC and descriptive records in collections for later replay and search. Social-network and YouTube captures add platform-specific extraction and media preservation. The README documents quality checks and indexing; their detailed runtime wiring is not shown in the excerpts, so those links are omitted or marked unverified.

Main entry points: [`webarc/worker.py`](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/worker.py) (Job worker), [`webarc/facebook.py`](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/facebook.py) (Social captures), [`webarc/redaction.py`](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/redaction.py) (Secret redaction), [`webarc/collections.py`](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/collections.py) (Collections)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Job management"]
  node_dashboard["Web dashboard<br/>[server.py]"]
  node_command["Command line<br/>[cli.py]"]
  node_worker["Job worker<br/>[worker.py]"]
  node_collections[("Collections<br/>[collections.py]")]
  node_config["Capture settings<br/>[config.py]"]
end

subgraph group_capture["Capture engines"]
  node_crawler["Automated crawl<br/>[crawler.py]"]
  node_recording["Session recording<br/>[recorder.py]"]
  node_browser["Browser capture<br/>[browser.py]"]
  node_social["Social captures<br/>[facebook.py]"]
  node_instagram["Instagram browser"]
  node_youtube["YouTube capture<br/>[youtube.py]"]
  node_x["X capture<br/>[x.py]"]
  node_instagram_browser["Instagram browser engine"]
end

subgraph group_preservation["Preservation records"]
  node_warc[("WARC archives<br/>[capture.py]")]
  node_redaction["Secret redaction<br/>[redaction.py]"]
  node_metadata["Descriptive metadata<br/>[metadata.py]"]
  node_manifest["Capture manifest"]
  node_settings_record["Settings provenance"]
  node_dedup[("Cross-job deduplication<br/>[dedup_index.py]")]
end

subgraph group_use["Archive access"]
  node_replay["Local replay<br/>[replay.py]"]
  node_indexing["Search indexing<br/>[indexer.py]"]
  node_warc_indexer["WARC indexer<br/>[WARCIndexer.java]"]
  node_quality["Quality checks<br/>[changes.py]"]
end

node_curator(("Curator"))

node_curator -->|"uses"| node_dashboard
node_curator -->|"runs"| node_command
node_dashboard -.->|"submits jobs"| node_worker
node_command -.->|"starts jobs"| node_worker
node_worker -->|"runs"| node_crawler
node_worker -->|"runs"| node_recording
node_worker -->|"runs"| node_social
node_worker -->|"runs"| node_instagram
node_worker -->|"runs"| node_youtube
node_worker -->|"runs"| node_x
node_worker -->|"uses"| node_collections
node_crawler -.->|"uses"| node_browser
node_recording -.->|"uses"| node_browser
node_instagram -->|"uses"| node_instagram_browser
node_social -->|"redacts data"| node_redaction
node_instagram -->|"redacts data"| node_redaction
node_x -->|"redacts data"| node_redaction
node_youtube -->|"redacts data"| node_redaction
node_collections -->|"uses index"| node_dedup
node_collections -->|"reads fields"| node_metadata
node_social -->|"records capture"| node_manifest
node_social -.->|"writes archive"| node_warc
node_metadata -.->|"embedded in"| node_warc
node_settings_record -->|"embedded in"| node_warc
node_curator -->|"browses locally"| node_replay
node_curator -->|"requests search"| node_indexing
node_indexing -.->|"uses"| node_warc_indexer
node_curator -->|"reviews"| node_quality

click node_dashboard "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/server.py"
click node_command "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/cli.py"
click node_worker "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/worker.py"
click node_collections "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/collections.py"
click node_crawler "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/crawler.py"
click node_recording "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/recorder.py"
click node_browser "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/browser.py"
click node_social "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/facebook.py"
click node_instagram "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/instagram_browser.py"
click node_youtube "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/youtube.py"
click node_x "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/x.py"
click node_config "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/config.py"
click node_warc "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/capture.py"
click node_redaction "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/redaction.py"
click node_metadata "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/metadata.py"
click node_manifest "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/capture_manifest.py"
click node_settings_record "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/capture_settings.py"
click node_dedup "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/dedup_index.py"
click node_replay "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/replay.py"
click node_indexing "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/indexer.py"
click node_warc_indexer "https://github.com/arifshaon/webcrawlmanager/blob/main/warc-indexer/src/main/java/uk/bl/wa/indexer/WARCIndexer.java"
click node_quality "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/changes.py"
click node_instagram_browser "https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/instagram_browser.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_command,node_worker,node_collections,node_config toneBlue
class node_crawler,node_recording,node_browser,node_social,node_instagram,node_youtube,node_x,node_instagram_browser toneAmber
class node_warc,node_redaction,node_metadata,node_manifest,node_settings_record,node_dedup toneMint
class node_replay,node_indexing,node_warc_indexer,node_quality toneRose
class node_curator toneIndigo
```

## Components

### Job management

- [Web dashboard](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/server.py): `webarc/server.py`
- [Command line](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/cli.py): `webarc/cli.py`
- [Job worker](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/worker.py): `webarc/worker.py`
- [Collections](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/collections.py): `webarc/collections.py`
- [Capture settings](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/config.py): `webarc/config.py`

### Capture engines

- [Automated crawl](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/crawler.py): `webarc/crawler.py`
- [Session recording](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/recorder.py): `webarc/recorder.py`
- [Browser capture](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/browser.py): `webarc/browser.py`
- [Social captures](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/facebook.py): `webarc/facebook.py`
- [Instagram browser](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/instagram_browser.py): `webarc/instagram_browser.py`
- [YouTube capture](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/youtube.py): `webarc/youtube.py`
- [X capture](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/x.py): `webarc/x.py`
- [Instagram browser engine](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/instagram_browser.py): `webarc/instagram_browser.py`

### Preservation records

- [WARC archives](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/capture.py): `webarc/capture.py`
- [Secret redaction](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/redaction.py): `webarc/redaction.py`
- [Descriptive metadata](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/metadata.py): `webarc/metadata.py`
- [Capture manifest](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/capture_manifest.py): `webarc/capture_manifest.py`
- [Settings provenance](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/capture_settings.py): `webarc/capture_settings.py`
- [Cross-job deduplication](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/dedup_index.py): `webarc/dedup_index.py`

### Archive access

- [Local replay](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/replay.py): `webarc/replay.py`
- [Search indexing](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/indexer.py): `webarc/indexer.py`
- [WARC indexer](https://github.com/arifshaon/webcrawlmanager/blob/main/warc-indexer/src/main/java/uk/bl/wa/indexer/WARCIndexer.java): `warc-indexer/src/main/java/uk/bl/wa/indexer/WARCIndexer.java`
- [Quality checks](https://github.com/arifshaon/webcrawlmanager/blob/main/webarc/changes.py): `webarc/changes.py`

### Other components

- **Curator**

## Connections

- Curator → Web dashboard: uses (evidence: `README.md`)
- Curator → Command line: runs (evidence: `README.md`)
- Web dashboard → Job worker: submits jobs
- Command line → Job worker: starts jobs
- Job worker → Automated crawl: runs (evidence: `webarc/worker.py`)
- Job worker → Session recording: runs (evidence: `webarc/worker.py`)
- Job worker → Social captures: runs (evidence: `webarc/worker.py`)
- Job worker → Instagram browser: runs (evidence: `webarc/worker.py`)
- Job worker → YouTube capture: runs (evidence: `webarc/worker.py`)
- Job worker → X capture: runs (evidence: `webarc/worker.py`)
- Job worker → Collections: uses (evidence: `webarc/worker.py`)
- Automated crawl → Browser capture: uses
- Session recording → Browser capture: uses
- Instagram browser → Instagram browser engine: uses (evidence: `webarc/instagram.py`)
- Social captures → Secret redaction: redacts data (evidence: `webarc/facebook.py`)
- Instagram browser → Secret redaction: redacts data (evidence: `webarc/instagram.py`)
- X capture → Secret redaction: redacts data (evidence: `webarc/x.py`)
- YouTube capture → Secret redaction: redacts data (evidence: `webarc/youtube.py`)
- Collections → Cross-job deduplication: uses index (evidence: `webarc/collections.py`)
- Collections → Descriptive metadata: reads fields (evidence: `webarc/collections.py`)
- Social captures → Capture manifest: records capture (evidence: `webarc/facebook.py`)
- Social captures → WARC archives: writes archive (evidence: `webarc/facebook.py`)
- Descriptive metadata → WARC archives: embedded in (evidence: `webarc/capture.py`)
- Settings provenance → WARC archives: embedded in (evidence: `webarc/capture_settings.py`)
- Curator → Local replay: browses locally (evidence: `README.md`)
- Curator → Search indexing: requests search (evidence: `README.md`)
- Search indexing → WARC indexer: uses
- Curator → Quality checks: reviews (evidence: `README.md`)