How umairahmedimran/script-jpop works
Project type and purpose
• This is a specialized automation tool for scraping and downloading torrent files from JPopsuki.eu.
• It combines a Playwright-based web scraper (sel.py) with a Scrapy-based downloader (myproject/).
• Purpose: log in, search, extract torrent metadata, store links, then batch-download .torrent files.
File structure
Top-level:
• sel.py: main Playwright scraper
• test_env.py: validates env vars
• requirements.txt, .env(.example): dependency and config templates
• download_links.txt: intermediate store of “link|name|size”
• downloads/: output folder for .torrent files
• myproject/: Scrapy project
• scrapy.cfg + nested myproject/ with settings.py, pipelines.py, spiders/download_spider.py
Architectural patterns:
• Two distinct modules (Playwright scraper → data store → Scrapy spider)
• Filesystem as message-bus between components
• Environment-driven configuration.
README for additional insights
• Two-step workflow: “Step 1: Scrape” then “Step 2: Download”
• Emphasis on env-variables for credentials and search parameters
• Configurable Scrapy settings and Playwright options
• Outputs: download_links.txt and downloads/ directory.
a. Main components:
• Config Loader (dotenv or env)
• Playwright Scraper (sel.py)
• Data Store (download_links.txt)
• Scrapy Downloader (download_spider.py inside myproject/)
• File Store (downloads/ directory)
• External Service (JPopsuki.eu)
b. Relationships and data flow:
1. Config Loader reads .env → provides credentials & search params to Playwright Scraper
2. Playwright Scraper logs into JPopsuki.eu → navigates search pages → extracts torrent metadata
3. Playwright writes results to download_links.txt
4. Scrapy Downloader reads download_links.txt → requests each torrent URL → downloads .torrent files
5. Scrapy writes .torrent files to downloads/
c. Architectural patterns & design principles:
• Modular separation: scraping vs downloading
• File-based decoupling (intermediate store)
• Config-driven behavior via environment variables
• Reusable Scrapy project scaffold
d. Key technologies & libraries:
• Playwright (Python) for browser automation
• Scrapy framework for crawling/downloading
• python-dotenv or os.getenv for config
• asyncio (under the hood for Playwright)
Detail
• Every major script, config file, and directory involved in the flow
• Optional behaviors (e.g., headless vs headed mode, adjustable Scrapy concurrency)