How smudgelord200/cb_web_scraper works
This is a scheduled-style event discovery script: running main.py initializes spaCy, gathers event listings from configured sites, filters for active Cate Blanchett involvement, compares matching URLs with notification history, saves a current-results snapshot, and emails newly found matches. The graph preserves static and dynamic fetching, parsing/relevance analysis, persistence, and reporting. README describes the broader capability set; source excerpts do not expose every scraper branch, so only evidenced call relationships are drawn.
Main entry points: main.py Run coordinator · data_storage.py JSON storage · scraper.py Site scraper · email_notification.py Email delivery
Components 15
Run orchestration
- Run coordinator
main.py - Email delivery
email_notification.py
Event discovery
- Site scraper
scraper.py - Static fetch
scraper.py - Event extraction
scraper.py - Involvement analysis
scraper.py - Browser fetching
selenium_scraper_methods.py
State and reporting
- JSON storage
data_storage.py - Current event log
- Notified URL history
- Recipient list
- Event console output
print_message.py
Other components
- Operator
- Email service
- Event websites
Connections 18
- Operator → Run coordinator: runs
README.md - Run coordinator → Site scraper: requests scan
main.py - Site scraper → Static fetch: fetches HTML
scraper.py - Site scraper → Browser fetching: requests browser HTML
scraper.py - Static fetch → Event websites: fetches pages
scraper.py - Browser fetching → Event websites: loads pages
selenium_scraper_methods.py - Site scraper → Event extraction: parses listings
scraper.py - Site scraper → Involvement analysis: checks involvement
scraper.py - Run coordinator → JSON storage: loads history
main.py - JSON storage → Notified URL history: reads URLs
data_storage.py - Run coordinator → JSON storage: saves snapshot
main.py - JSON storage → Current event log: writes events
data_storage.py - Run coordinator → JSON storage: loads recipients
main.py - JSON storage → Recipient list: reads list
data_storage.py - Run coordinator → JSON storage: updates history
main.py - Run coordinator → Event console output: prints events
main.py - Run coordinator → Email delivery: sends new matches
main.py - Email delivery → Email service: delivers email
email_notification.py