How hero0p/scholarship-crawler works
Atlas is an evidence-first scholarship intelligence crawler: it finds candidate leads, resolves them to official sources, crawls and classifies pages, extracts traceable fields, verifies records, and tracks changes and lifecycle status. Its main runtime entry point is the CLI; the dashboard presents results, while Wayback baselines, exports, audits, and adaptive scheduling support ongoing use. The README documents the overall workflow; source-level excerpts are partial, so detailed internal calls and some dashboard wiring are not asserted.
Main entry points: cli.py Atlas CLI · process.py Record Processing · spiders Crawler Spiders · session.py Database Session
Components 26
Discovery and Crawling
- Crawler Spiders
crawler/atlas_crawler/spiders - Polite Middleware
crawler/atlas_crawler/middlewares.py - Crawler Pipeline
crawler/atlas_crawler/pipelines.py - Lead Resolver
atlas/discovery/resolver.py
Page Processing
- Source Classifier
atlas/classify/source_classifier.py - Page Classifier
atlas/classify/page_classifier.py - Content Converter
atlas/extract/content.py - Section Segmentation
atlas/extract/sections.py - Field Extractors
atlas/extract/fields.py - Listing Cards
atlas/extract/cards.py - Record Assembly
atlas/extract/record.py - Record Processing
atlas/process.py
Verification and History
- Confidence Verifier
atlas/verify/confidence.py - Lifecycle Status
atlas/lifecycle/status.py - Wayback Baselines
atlas/discovery/wayback.py - Recrawl Policy
atlas/schedule/frequency.py
Evidence and Storage
- Immutable Snapshots
atlas/snapshots.py - Field Evidence
atlas/extract/evidence.py - Record Models
atlas/db/models.py - Database Session
atlas/db/session.py
Operations and Access
- Atlas CLI
atlas/cli.py - Scholarship Dashboard
dashboard/app.py
Other components
- Operator
- Official Scholarship Sites
- Discovery Aggregators
- Internet Archive
Connections 28
- Operator → Atlas CLI: runs commands
README.md - Atlas CLI → Crawler Spiders: starts crawl
README.md - Crawler Spiders → Discovery Aggregators: mines leads
README.md - Crawler Spiders → Official Scholarship Sites: crawls sources
README.md - Crawler Spiders → Polite Middleware: uses middleware
crawler/atlas_crawler/spiders/base.py - Crawler Pipeline → Immutable Snapshots: stores fetches
crawler/atlas_crawler/pipelines.py - Atlas CLI → Lead Resolver: resolves leads
README.md - Lead Resolver → Source Classifier: classifies domains
atlas/discovery/resolver.py - Atlas CLI → Record Processing: processes pages
README.md - Record Processing → Immutable Snapshots: reads snapshots
atlas/process.py - Record Processing → Content Converter: converts content
atlas/process.py - Content Converter → Section Segmentation: segments text
atlas/extract/content.py - Record Processing → Page Classifier: classifies pages
atlas/process.py - Page Classifier → Field Extractors: uses field signals
atlas/classify/page_classifier.py - Record Processing → Listing Cards: extracts listings
atlas/process.py - Record Processing → Record Assembly: builds records
atlas/process.py - Record Assembly → Field Extractors: extracts fields
atlas/extract/record.py - Record Processing → Field Evidence: records evidence
atlas/process.py - Record Processing → Record Models: updates records
atlas/process.py - Atlas CLI → Confidence Verifier: verifies records
README.md - Atlas CLI → Lifecycle Status: updates status
README.md - Atlas CLI → Wayback Baselines: runs baselines
README.md - Wayback Baselines → Internet Archive: fetches history
atlas/discovery/wayback.py - Atlas CLI → Recrawl Policy: schedules recrawls
atlas/cli.py - Database Session → Record Models: initializes schema
atlas/db/session.py - Atlas CLI → Database Session: opens database
atlas/cli.py - Operator → Scholarship Dashboard: views results
README.md - Scholarship Dashboard → Database Session