# kriai/high-signal-dashboard architecture

> Architecture diagram of the GitHub repository kriai/high-signal-dashboard, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kriai/high-signal-dashboard
- Repository: https://github.com/kriai/high-signal-dashboard

## Overview

High Signal collects and scores headlines from configured feeds and publisher pages, publishes snapshots, and presents them in a browser dashboard. The main evidenced workflow is a scheduled scrape: the job loads source configuration and prior state, runs the scraper and pipeline, then publishes through D1; the Flask app also provides dashboard/API and source-management functionality. Source discovery and testing run as D1-backed jobs. The Worker and browser dashboard are present, but their interaction is not established by the supplied excerpts, so that connection is omitted.

Main entry points: [`pipeline.py`](https://github.com/kriai/high-signal-dashboard/blob/main/pipeline.py) (Scrape Pipeline), [`app.py`](https://github.com/kriai/high-signal-dashboard/blob/main/app.py) (Flask API), [`scraper.py`](https://github.com/kriai/high-signal-dashboard/blob/main/scraper.py) (Feed Scraper), [`d1_store.py`](https://github.com/kriai/high-signal-dashboard/blob/main/d1_store.py) (D1 Publication Store)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection and Processing"]
  node_scrape_job["Scheduled Scrape<br/>[scrape_job.py]"]
  node_scraper["Feed Scraper<br/>[scraper.py]"]
  node_source_config["Source Configuration<br/>[source_config.py]"]
  node_pipeline["Scrape Pipeline<br/>[pipeline.py]"]
end

subgraph group_publishing["Publication and State"]
  node_d1_store["D1 Publication Store<br/>[d1_store.py]"]
  node_d1[("Cloudflare D1")]
  node_legacy_store[("Local / Blob Store<br/>[store.py]")]
  node_state_import["State Importer<br/>[import_state.py]"]
end

subgraph group_source_admin["Source Administration"]
  node_source_tools["Source Discovery and Tests<br/>[source_tools.py]"]
  node_source_tool_job["Source Tool Jobs<br/>[source_tool_job.py]"]
end

subgraph group_web["Web Experience"]
  node_flask_app["Flask API<br/>[app.py]"]
  node_dashboard["Dashboard Client<br/>[dashboard.js]"]
  node_dashboard_template["Dashboard Template<br/>[index.html]"]
  node_worker["Cloudflare Worker<br/>[index.ts]"]
end

node_scheduler(("GitHub Actions"))

node_scheduler -->|"schedules"| node_scrape_job
node_scrape_job -->|"runs"| node_pipeline
node_scrape_job -->|"loads"| node_scraper
node_scrape_job -->|"publishes through"| node_d1_store
node_pipeline -->|"scrapes with"| node_scraper
node_pipeline -->|"uses"| node_source_config
node_pipeline -->|"prepares publication"| node_d1_store
node_scraper -->|"normalizes sources"| node_source_config
node_d1_store -->|"queries"| node_d1
node_legacy_store -->|"supports D1 backend"| node_d1
node_state_import -->|"imports state"| node_d1_store
node_state_import -->|"prepares publication"| node_pipeline
node_flask_app -->|"uses"| node_scraper
node_flask_app -->|"uses"| node_pipeline
node_flask_app -->|"reads and writes"| node_legacy_store
node_flask_app -->|"uses"| node_source_config
node_flask_app -->|"delegates source tools"| node_source_tools
node_source_tool_job -->|"runs"| node_source_tools
node_source_tool_job -->|"creates"| node_scraper
node_source_tool_job -->|"claims jobs through"| node_d1_store
node_dashboard_template -.->|"hosts client"| node_dashboard

click node_scrape_job "https://github.com/kriai/high-signal-dashboard/blob/main/scrape_job.py"
click node_scraper "https://github.com/kriai/high-signal-dashboard/blob/main/scraper.py"
click node_source_config "https://github.com/kriai/high-signal-dashboard/blob/main/source_config.py"
click node_pipeline "https://github.com/kriai/high-signal-dashboard/blob/main/pipeline.py"
click node_d1_store "https://github.com/kriai/high-signal-dashboard/blob/main/d1_store.py"
click node_legacy_store "https://github.com/kriai/high-signal-dashboard/blob/main/store.py"
click node_state_import "https://github.com/kriai/high-signal-dashboard/blob/main/scripts/import_state.py"
click node_flask_app "https://github.com/kriai/high-signal-dashboard/blob/main/app.py"
click node_dashboard "https://github.com/kriai/high-signal-dashboard/blob/main/static/js/dashboard.js"
click node_dashboard_template "https://github.com/kriai/high-signal-dashboard/blob/main/templates/index.html"
click node_worker "https://github.com/kriai/high-signal-dashboard/blob/main/worker/src/index.ts"
click node_source_tools "https://github.com/kriai/high-signal-dashboard/blob/main/source_tools.py"
click node_source_tool_job "https://github.com/kriai/high-signal-dashboard/blob/main/source_tool_job.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scrape_job,node_scraper,node_source_config,node_pipeline toneBlue
class node_d1_store,node_d1,node_legacy_store,node_state_import toneAmber
class node_source_tools,node_source_tool_job toneMint
class node_flask_app,node_dashboard,node_dashboard_template,node_worker toneRose
class node_scheduler toneIndigo
```

## Components

### Collection and Processing

- [Scheduled Scrape](https://github.com/kriai/high-signal-dashboard/blob/main/scrape_job.py): `scrape_job.py`
- [Feed Scraper](https://github.com/kriai/high-signal-dashboard/blob/main/scraper.py): `scraper.py`
- [Source Configuration](https://github.com/kriai/high-signal-dashboard/blob/main/source_config.py): `source_config.py`
- [Scrape Pipeline](https://github.com/kriai/high-signal-dashboard/blob/main/pipeline.py): `pipeline.py`

### Publication and State

- [D1 Publication Store](https://github.com/kriai/high-signal-dashboard/blob/main/d1_store.py): `d1_store.py`
- **Cloudflare D1**
- [Local / Blob Store](https://github.com/kriai/high-signal-dashboard/blob/main/store.py): `store.py`
- [State Importer](https://github.com/kriai/high-signal-dashboard/blob/main/scripts/import_state.py): `scripts/import_state.py`

### Source Administration

- [Source Discovery and Tests](https://github.com/kriai/high-signal-dashboard/blob/main/source_tools.py): `source_tools.py`
- [Source Tool Jobs](https://github.com/kriai/high-signal-dashboard/blob/main/source_tool_job.py): `source_tool_job.py`

### Web Experience

- [Flask API](https://github.com/kriai/high-signal-dashboard/blob/main/app.py): `app.py`
- [Dashboard Client](https://github.com/kriai/high-signal-dashboard/blob/main/static/js/dashboard.js): `static/js/dashboard.js`
- [Dashboard Template](https://github.com/kriai/high-signal-dashboard/blob/main/templates/index.html): `templates/index.html`
- [Cloudflare Worker](https://github.com/kriai/high-signal-dashboard/blob/main/worker/src/index.ts): `worker/src/index.ts`

### Other components

- **GitHub Actions**

## Connections

- GitHub Actions → Scheduled Scrape: schedules (evidence: `scrape_job.py`)
- Scheduled Scrape → Scrape Pipeline: runs (evidence: `scrape_job.py`)
- Scheduled Scrape → Feed Scraper: loads (evidence: `scrape_job.py`)
- Scheduled Scrape → D1 Publication Store: publishes through (evidence: `scrape_job.py`)
- Scrape Pipeline → Feed Scraper: scrapes with (evidence: `pipeline.py`)
- Scrape Pipeline → Source Configuration: uses (evidence: `pipeline.py`)
- Scrape Pipeline → D1 Publication Store: prepares publication (evidence: `pipeline.py`)
- Feed Scraper → Source Configuration: normalizes sources (evidence: `scraper.py`)
- D1 Publication Store → Cloudflare D1: queries (evidence: `d1_store.py`)
- Local / Blob Store → Cloudflare D1: supports D1 backend (evidence: `store.py`)
- State Importer → D1 Publication Store: imports state (evidence: `scripts/import_state.py`)
- State Importer → Scrape Pipeline: prepares publication (evidence: `scripts/import_state.py`)
- Flask API → Feed Scraper: uses (evidence: `app.py`)
- Flask API → Scrape Pipeline: uses (evidence: `app.py`)
- Flask API → Local / Blob Store: reads and writes (evidence: `app.py`)
- Flask API → Source Configuration: uses (evidence: `app.py`)
- Flask API → Source Discovery and Tests: delegates source tools (evidence: `app.py`)
- Source Tool Jobs → Source Discovery and Tests: runs (evidence: `source_tool_job.py`)
- Source Tool Jobs → Feed Scraper: creates (evidence: `source_tool_job.py`)
- Source Tool Jobs → D1 Publication Store: claims jobs through (evidence: `source_tool_job.py`)
- Dashboard Template → Dashboard Client: hosts client