How kennethlucia/scrappy-project works
This is a small configurable Scrapy crawler. Its workflow is: the manager configures and starts Scrapy, which runs the spider against starting URLs; the spider extracts page data, optionally follows links, and writes collected results to JSON when it closes. The single source file shows the manager-to-spider setup and spider behavior; Scrapy’s internal request scheduling is represented only at the framework boundary.
Main entry points: Scrappy_Scripts.py Web crawler spider
Components 6
Crawler runtime
- Crawler manager
Scrappy_Scripts.py - Scrapy process
- Web crawler spider
Scrappy_Scripts.py - JSON output
Scrappy_Scripts.py
Other components
- Caller
- Target websites
Connections 6
- Caller → Crawler manager: runs example
- Crawler manager → Scrapy process: starts process
Scrappy_Scripts.py - Scrapy process → Web crawler spider: schedules spider
Scrappy_Scripts.py - Web crawler spider → Target websites: requests pages
Scrappy_Scripts.py - Web crawler spider → Web crawler spider: follows links
Scrappy_Scripts.py - Web crawler spider → JSON output: writes results
Scrappy_Scripts.py