# ravennova1/channelscraper architecture

> Architecture diagram of the GitHub repository ravennova1/channelscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ravennova1/channelscraper
- Repository: https://github.com/ravennova1/channelscraper

## Overview

ChannelScraper turns a YouTube channel, playlist, or video into timestamped transcripts and a standalone offline search page. The main workflow can start in the browser dashboard or run as a pipeline: discover videos, fetch captions, align words and timestamps, compile transcript files, then build the search page. Separate capabilities update existing workspaces, resume interrupted processing, and detect duplicate or overlapping videos. The README documents the pipeline stages; the sampled source confirms key orchestration and compilation links, but does not expose every stage implementation or dashboard route.

Main entry points: [`core/pipeline.py`](https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py) (Pipeline Orchestrator), [`core/build_search.py`](https://github.com/ravennova1/channelscraper/blob/main/core/build_search.py) (Search Compiler), [`runs`](https://github.com/ravennova1/channelscraper/tree/main/runs) (Transcript Files), [`core/continue_run.py`](https://github.com/ravennova1/channelscraper/blob/main/core/continue_run.py) (Run Recovery)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["Dashboard"]
  node_dashboard_ui["Dashboard UI<br/>[index.html]"]
  node_dashboard_server["Dashboard Server<br/>[app.py]"]
end

subgraph group_transcription["Transcription Pipeline"]
  node_pipeline["Pipeline Orchestrator<br/>[pipeline.py]"]
  node_discovery["Video Discovery<br/>[pipeline.py]"]
  node_caption_fetch["Caption Fetching<br/>[pipeline.py]"]
  node_word_timing["Word Timing<br/>[pipeline.py]"]
  node_transcript_compile["Transcript Compilation<br/>[pipeline.py]"]
  node_transcript_files["Transcript Files"]
end

subgraph group_search["Offline Search"]
  node_search_builder["Search Compiler<br/>[build_search.py]"]
  node_offline_search["Offline Search Page<br/>[build_search.py]"]
end

subgraph group_operations["Workspace Operations"]
  node_incremental_update["Incremental Updater<br/>[update_channel.py]"]
  node_resume_run["Run Recovery<br/>[continue_run.py]"]
  node_duplicate_detection["Duplicate Detection<br/>[find_duplicates.py]"]
  node_duplicate_report["Duplicate Report<br/>[find_duplicates.py]"]
end

node_user(("User"))
node_youtube{{"YouTube"}}
node_browser(("Browser"))

node_user -->|"uses"| node_dashboard_ui
node_dashboard_ui -.->|"communicates with"| node_dashboard_server
node_dashboard_server -->|"starts pipeline"| node_pipeline
node_pipeline -->|"runs stage"| node_discovery
node_discovery -->|"lists videos"| node_youtube
node_pipeline -->|"runs stage"| node_caption_fetch
node_caption_fetch -->|"fetches captions"| node_youtube
node_pipeline -->|"runs stage"| node_word_timing
node_pipeline -->|"runs stage"| node_transcript_compile
node_pipeline -->|"writes"| node_transcript_files
node_pipeline -->|"calls"| node_search_builder
node_transcript_compile -->|"produces"| node_transcript_files
node_search_builder -->|"reads"| node_transcript_files
node_search_builder -->|"generates"| node_offline_search
node_user -->|"requests update"| node_incremental_update
node_incremental_update -->|"uses workspace engine"| node_pipeline
node_user -->|"resumes run"| node_resume_run
node_resume_run -->|"uses workspace engine"| node_pipeline
node_resume_run -->|"builds search page"| node_search_builder
node_user -->|"runs analysis"| node_duplicate_detection
node_duplicate_detection -->|"reads transcript data"| node_transcript_files
node_duplicate_detection -->|"exports report"| node_duplicate_report
node_user -->|"opens page"| node_browser
node_browser -->|"searches locally"| node_offline_search

click node_dashboard_ui "https://github.com/ravennova1/channelscraper/blob/main/core/ui/index.html"
click node_dashboard_server "https://github.com/ravennova1/channelscraper/blob/main/core/app.py"
click node_pipeline "https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py"
click node_discovery "https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py"
click node_caption_fetch "https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py"
click node_word_timing "https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py"
click node_transcript_compile "https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py"
click node_transcript_files "https://github.com/ravennova1/channelscraper/tree/main/runs"
click node_search_builder "https://github.com/ravennova1/channelscraper/blob/main/core/build_search.py"
click node_offline_search "https://github.com/ravennova1/channelscraper/blob/main/core/build_search.py"
click node_incremental_update "https://github.com/ravennova1/channelscraper/blob/main/core/update_channel.py"
click node_resume_run "https://github.com/ravennova1/channelscraper/blob/main/core/continue_run.py"
click node_duplicate_detection "https://github.com/ravennova1/channelscraper/blob/main/core/find_duplicates.py"
click node_duplicate_report "https://github.com/ravennova1/channelscraper/blob/main/core/find_duplicates.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard_ui,node_dashboard_server,node_user,node_browser toneBlue
class node_pipeline,node_discovery,node_caption_fetch,node_word_timing,node_transcript_compile,node_transcript_files toneAmber
class node_search_builder,node_offline_search toneMint
class node_incremental_update,node_resume_run,node_duplicate_detection,node_duplicate_report toneRose
class node_youtube toneIndigo
```

## Components

### Dashboard

- [Dashboard UI](https://github.com/ravennova1/channelscraper/blob/main/core/ui/index.html): `core/ui/index.html`
- [Dashboard Server](https://github.com/ravennova1/channelscraper/blob/main/core/app.py): `core/app.py`

### Transcription Pipeline

- [Pipeline Orchestrator](https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py): `core/pipeline.py`
- [Video Discovery](https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py): `core/pipeline.py`
- [Caption Fetching](https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py): `core/pipeline.py`
- [Word Timing](https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py): `core/pipeline.py`
- [Transcript Compilation](https://github.com/ravennova1/channelscraper/blob/main/core/pipeline.py): `core/pipeline.py`
- [Transcript Files](https://github.com/ravennova1/channelscraper/tree/main/runs): `runs`

### Offline Search

- [Search Compiler](https://github.com/ravennova1/channelscraper/blob/main/core/build_search.py): `core/build_search.py`
- [Offline Search Page](https://github.com/ravennova1/channelscraper/blob/main/core/build_search.py): `core/build_search.py`

### Workspace Operations

- [Incremental Updater](https://github.com/ravennova1/channelscraper/blob/main/core/update_channel.py): `core/update_channel.py`
- [Run Recovery](https://github.com/ravennova1/channelscraper/blob/main/core/continue_run.py): `core/continue_run.py`
- [Duplicate Detection](https://github.com/ravennova1/channelscraper/blob/main/core/find_duplicates.py): `core/find_duplicates.py`
- [Duplicate Report](https://github.com/ravennova1/channelscraper/blob/main/core/find_duplicates.py): `core/find_duplicates.py`

### Other components

- **User**
- **YouTube**
- **Browser**

## Connections

- User → Dashboard UI: uses (evidence: `README.md`)
- Dashboard UI → Dashboard Server: communicates with
- Dashboard Server → Pipeline Orchestrator: starts pipeline (evidence: `core/app.py`)
- Pipeline Orchestrator → Video Discovery: runs stage (evidence: `README.md`)
- Video Discovery → YouTube: lists videos (evidence: `README.md`)
- Pipeline Orchestrator → Caption Fetching: runs stage (evidence: `README.md`)
- Caption Fetching → YouTube: fetches captions (evidence: `README.md`)
- Pipeline Orchestrator → Word Timing: runs stage (evidence: `README.md`)
- Pipeline Orchestrator → Transcript Compilation: runs stage (evidence: `README.md`)
- Pipeline Orchestrator → Transcript Files: writes (evidence: `README.md`)
- Pipeline Orchestrator → Search Compiler: calls (evidence: `core/pipeline.py`)
- Transcript Compilation → Transcript Files: produces (evidence: `README.md`)
- Search Compiler → Transcript Files: reads (evidence: `README.md`)
- Search Compiler → Offline Search Page: generates (evidence: `core/build_search.py`)
- User → Incremental Updater: requests update (evidence: `README.md`)
- Incremental Updater → Pipeline Orchestrator: uses workspace engine (evidence: `core/update_channel.py`)
- User → Run Recovery: resumes run (evidence: `README.md`)
- Run Recovery → Pipeline Orchestrator: uses workspace engine (evidence: `core/continue_run.py`)
- Run Recovery → Search Compiler: builds search page (evidence: `core/continue_run.py`)
- User → Duplicate Detection: runs analysis (evidence: `README.md`)
- Duplicate Detection → Transcript Files: reads transcript data (evidence: `core/find_duplicates.py`)
- Duplicate Detection → Duplicate Report: exports report (evidence: `core/find_duplicates.py`)
- User → Browser: opens page (evidence: `README.md`)
- Browser → Offline Search Page: searches locally (evidence: `README.md`)
