# hero0p/scholarship-crawler architecture

> Architecture diagram of the GitHub repository hero0p/scholarship-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hero0p/scholarship-crawler
- Repository: https://github.com/hero0p/scholarship-crawler

## Overview

Atlas is an evidence-first scholarship intelligence crawler: it finds candidate leads, resolves them to official sources, crawls and classifies pages, extracts traceable fields, verifies records, and tracks changes and lifecycle status. Its main runtime entry point is the CLI; the dashboard presents results, while Wayback baselines, exports, audits, and adaptive scheduling support ongoing use. The README documents the overall workflow; source-level excerpts are partial, so detailed internal calls and some dashboard wiring are not asserted.

Main entry points: [`atlas/cli.py`](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/cli.py) (Atlas CLI), [`atlas/process.py`](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/process.py) (Record Processing), [`crawler/atlas_crawler/spiders`](https://github.com/hero0p/scholarship-crawler/tree/main/crawler/atlas_crawler/spiders) (Crawler Spiders), [`atlas/db/session.py`](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/db/session.py) (Database Session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_intake["Discovery and Crawling"]
  node_crawler["Crawler Spiders"]
  node_middlewares["Polite Middleware<br/>[middlewares.py]"]
  node_crawlerpipe["Crawler Pipeline<br/>[pipelines.py]"]
  node_leads["Lead Resolver<br/>[resolver.py]"]
end

subgraph group_processing["Page Processing"]
  node_sourceclass["Source Classifier"]
  node_pageclass["Page Classifier<br/>[page_classifier.py]"]
  node_content["Content Converter<br/>[content.py]"]
  node_sections["Section Segmentation<br/>[sections.py]"]
  node_fields["Field Extractors<br/>[fields.py]"]
  node_cards["Listing Cards<br/>[cards.py]"]
  node_record["Record Assembly<br/>[record.py]"]
  node_process["Record Processing<br/>[process.py]"]
end

subgraph group_trust["Verification and History"]
  node_verify["Confidence Verifier<br/>[confidence.py]"]
  node_lifecycle["Lifecycle Status<br/>[status.py]"]
  node_wayback["Wayback Baselines<br/>[wayback.py]"]
  node_frequency["Recrawl Policy<br/>[frequency.py]"]
end

subgraph group_persistence["Evidence and Storage"]
  node_snapshots["Immutable Snapshots<br/>[snapshots.py]"]
  node_evidence["Field Evidence<br/>[evidence.py]"]
  node_dbmodels["Record Models<br/>[models.py]"]
  node_dbsession["Database Session<br/>[session.py]"]
end

subgraph group_access["Operations and Access"]
  node_cli["Atlas CLI<br/>[cli.py]"]
  node_dashboard["Scholarship Dashboard<br/>[app.py]"]
end

node_user(("Operator"))
node_official{{"Official Scholarship Sites"}}
node_aggregators{{"Discovery Aggregators"}}
node_archive{{"Internet Archive"}}

node_user -->|"runs commands"| node_cli
node_cli -->|"starts crawl"| node_crawler
node_crawler -->|"mines leads"| node_aggregators
node_crawler -->|"crawls sources"| node_official
node_crawler -->|"uses middleware"| node_middlewares
node_crawlerpipe -->|"stores fetches"| node_snapshots
node_cli -->|"resolves leads"| node_leads
node_leads -->|"classifies domains"| node_sourceclass
node_cli -->|"processes pages"| node_process
node_process -->|"reads snapshots"| node_snapshots
node_process -->|"converts content"| node_content
node_content -->|"segments text"| node_sections
node_process -->|"classifies pages"| node_pageclass
node_pageclass -->|"uses field signals"| node_fields
node_process -->|"extracts listings"| node_cards
node_process -->|"builds records"| node_record
node_record -->|"extracts fields"| node_fields
node_process -->|"records evidence"| node_evidence
node_process -->|"updates records"| node_dbmodels
node_cli -->|"verifies records"| node_verify
node_cli -->|"updates status"| node_lifecycle
node_cli -->|"runs baselines"| node_wayback
node_wayback -->|"fetches history"| node_archive
node_cli -->|"schedules recrawls"| node_frequency
node_dbsession -->|"initializes schema"| node_dbmodels
node_cli -->|"opens database"| node_dbsession
node_user -->|"views results"| node_dashboard
node_dashboard -.-> node_dbsession

click node_cli "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/cli.py"
click node_crawler "https://github.com/hero0p/scholarship-crawler/tree/main/crawler/atlas_crawler/spiders"
click node_middlewares "https://github.com/hero0p/scholarship-crawler/blob/main/crawler/atlas_crawler/middlewares.py"
click node_crawlerpipe "https://github.com/hero0p/scholarship-crawler/blob/main/crawler/atlas_crawler/pipelines.py"
click node_leads "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/discovery/resolver.py"
click node_sourceclass "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/classify/source_classifier.py"
click node_pageclass "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/classify/page_classifier.py"
click node_content "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/content.py"
click node_sections "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/sections.py"
click node_fields "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/fields.py"
click node_cards "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/cards.py"
click node_record "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/record.py"
click node_process "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/process.py"
click node_verify "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/verify/confidence.py"
click node_lifecycle "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/lifecycle/status.py"
click node_wayback "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/discovery/wayback.py"
click node_frequency "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/schedule/frequency.py"
click node_snapshots "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/snapshots.py"
click node_evidence "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/evidence.py"
click node_dbmodels "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/db/models.py"
click node_dbsession "https://github.com/hero0p/scholarship-crawler/blob/main/atlas/db/session.py"
click node_dashboard "https://github.com/hero0p/scholarship-crawler/blob/main/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawler,node_middlewares,node_crawlerpipe,node_leads toneBlue
class node_sourceclass,node_pageclass,node_content,node_sections,node_fields,node_cards,node_record,node_process toneAmber
class node_verify,node_lifecycle,node_wayback,node_frequency toneMint
class node_snapshots,node_evidence,node_dbmodels,node_dbsession toneRose
class node_cli,node_dashboard,node_user,node_official,node_aggregators,node_archive toneIndigo
```

## Components

### Discovery and Crawling

- [Crawler Spiders](https://github.com/hero0p/scholarship-crawler/tree/main/crawler/atlas_crawler/spiders): `crawler/atlas_crawler/spiders`
- [Polite Middleware](https://github.com/hero0p/scholarship-crawler/blob/main/crawler/atlas_crawler/middlewares.py): `crawler/atlas_crawler/middlewares.py`
- [Crawler Pipeline](https://github.com/hero0p/scholarship-crawler/blob/main/crawler/atlas_crawler/pipelines.py): `crawler/atlas_crawler/pipelines.py`
- [Lead Resolver](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/discovery/resolver.py): `atlas/discovery/resolver.py`

### Page Processing

- [Source Classifier](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/classify/source_classifier.py): `atlas/classify/source_classifier.py`
- [Page Classifier](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/classify/page_classifier.py): `atlas/classify/page_classifier.py`
- [Content Converter](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/content.py): `atlas/extract/content.py`
- [Section Segmentation](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/sections.py): `atlas/extract/sections.py`
- [Field Extractors](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/fields.py): `atlas/extract/fields.py`
- [Listing Cards](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/cards.py): `atlas/extract/cards.py`
- [Record Assembly](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/record.py): `atlas/extract/record.py`
- [Record Processing](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/process.py): `atlas/process.py`

### Verification and History

- [Confidence Verifier](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/verify/confidence.py): `atlas/verify/confidence.py`
- [Lifecycle Status](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/lifecycle/status.py): `atlas/lifecycle/status.py`
- [Wayback Baselines](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/discovery/wayback.py): `atlas/discovery/wayback.py`
- [Recrawl Policy](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/schedule/frequency.py): `atlas/schedule/frequency.py`

### Evidence and Storage

- [Immutable Snapshots](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/snapshots.py): `atlas/snapshots.py`
- [Field Evidence](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/extract/evidence.py): `atlas/extract/evidence.py`
- [Record Models](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/db/models.py): `atlas/db/models.py`
- [Database Session](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/db/session.py): `atlas/db/session.py`

### Operations and Access

- [Atlas CLI](https://github.com/hero0p/scholarship-crawler/blob/main/atlas/cli.py): `atlas/cli.py`
- [Scholarship Dashboard](https://github.com/hero0p/scholarship-crawler/blob/main/dashboard/app.py): `dashboard/app.py`

### Other components

- **Operator**
- **Official Scholarship Sites**
- **Discovery Aggregators**
- **Internet Archive**

## Connections

- Operator → Atlas CLI: runs commands (evidence: `README.md`)
- Atlas CLI → Crawler Spiders: starts crawl (evidence: `README.md`)
- Crawler Spiders → Discovery Aggregators: mines leads (evidence: `README.md`)
- Crawler Spiders → Official Scholarship Sites: crawls sources (evidence: `README.md`)
- Crawler Spiders → Polite Middleware: uses middleware (evidence: `crawler/atlas_crawler/spiders/base.py`)
- Crawler Pipeline → Immutable Snapshots: stores fetches (evidence: `crawler/atlas_crawler/pipelines.py`)
- Atlas CLI → Lead Resolver: resolves leads (evidence: `README.md`)
- Lead Resolver → Source Classifier: classifies domains (evidence: `atlas/discovery/resolver.py`)
- Atlas CLI → Record Processing: processes pages (evidence: `README.md`)
- Record Processing → Immutable Snapshots: reads snapshots (evidence: `atlas/process.py`)
- Record Processing → Content Converter: converts content (evidence: `atlas/process.py`)
- Content Converter → Section Segmentation: segments text (evidence: `atlas/extract/content.py`)
- Record Processing → Page Classifier: classifies pages (evidence: `atlas/process.py`)
- Page Classifier → Field Extractors: uses field signals (evidence: `atlas/classify/page_classifier.py`)
- Record Processing → Listing Cards: extracts listings (evidence: `atlas/process.py`)
- Record Processing → Record Assembly: builds records (evidence: `atlas/process.py`)
- Record Assembly → Field Extractors: extracts fields (evidence: `atlas/extract/record.py`)
- Record Processing → Field Evidence: records evidence (evidence: `atlas/process.py`)
- Record Processing → Record Models: updates records (evidence: `atlas/process.py`)
- Atlas CLI → Confidence Verifier: verifies records (evidence: `README.md`)
- Atlas CLI → Lifecycle Status: updates status (evidence: `README.md`)
- Atlas CLI → Wayback Baselines: runs baselines (evidence: `README.md`)
- Wayback Baselines → Internet Archive: fetches history (evidence: `atlas/discovery/wayback.py`)
- Atlas CLI → Recrawl Policy: schedules recrawls (evidence: `atlas/cli.py`)
- Database Session → Record Models: initializes schema (evidence: `atlas/db/session.py`)
- Atlas CLI → Database Session: opens database (evidence: `atlas/cli.py`)
- Operator → Scholarship Dashboard: views results (evidence: `README.md`)
- Scholarship Dashboard → Database Session
