# umairahmedimran/script-jpop architecture

> Architecture diagram of the GitHub repository umairahmedimran/script-jpop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/umairahmedimran/script-jpop
- Repository: https://github.com/umairahmedimran/script-jpop

## Overview

##### Project type and purpose
- This is a specialized automation tool for scraping and downloading torrent files from JPopsuki.eu.
- It combines a Playwright-based web scraper (sel.py) with a Scrapy-based downloader (myproject/).
- Purpose: log in, search, extract torrent metadata, store links, then batch-download .torrent files.

##### File structure
Top-level:
- sel.py: main Playwright scraper
- test_env.py: validates env vars
- requirements.txt, .env(.example): dependency and config templates
- download_links.txt: intermediate store of “link|name|size”
- downloads/: output folder for .torrent files
- myproject/: Scrapy project
- scrapy.cfg + nested myproject/ with settings.py, pipelines.py, spiders/download_spider.py

Architectural patterns:
- Two distinct modules (Playwright scraper → data store → Scrapy spider)
- Filesystem as message-bus between components
- Environment-driven configuration.

##### README for additional insights
- Two-step workflow: “Step 1: Scrape” then “Step 2: Download”
- Emphasis on env-variables for credentials and search parameters
- Configurable Scrapy settings and Playwright options
- Outputs: download_links.txt and downloads/ directory.

a. Main components:
- Config Loader (dotenv or env)
- Playwright Scraper (sel.py)
- Data Store (download_links.txt)
- Scrapy Downloader (download_spider.py inside myproject/)
- File Store (downloads/ directory)
- External Service (JPopsuki.eu)
b. Relationships and data flow:
1. Config Loader reads .env → provides credentials & search params to Playwright Scraper
2. Playwright Scraper logs into JPopsuki.eu → navigates search pages → extracts torrent metadata
3. Playwright writes results to download_links.txt
4. Scrapy Downloader reads download_links.txt → requests each torrent URL → downloads .torrent files
5. Scrapy writes .torrent files to downloads/
c. Architectural patterns & design principles:
- Modular separation: scraping vs downloading
- File-based decoupling (intermediate store)
- Config-driven behavior via environment variables
- Reusable Scrapy project scaffold
d. Key technologies & libraries:
- Playwright (Python) for browser automation
- Scrapy framework for crawling/downloading
- python-dotenv or os.getenv for config
- asyncio (under the hood for Playwright)

##### Detail
- Every major script, config file, and directory involved in the flow
- Optional behaviors (e.g., headless vs headed mode, adjustable Scrapy concurrency)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Service
    JPopsuki["JPopsuki.eu"]:::external

    %% Config Loader Subgraph
    subgraph "Config Loader"
        direction TB
        EnvExample[".env.example"]:::config
        Env[".env"]:::config
        TestEnv["test_env.py"]:::config
    end

    %% Playwright Scraper
    Sel["Playwright Scraper (sel.py)"]:::scraper

    %% Data Store
    Links["download_links.txt"]:::datastore

    %% Scrapy Project Subgraph
    subgraph "Scrapy Project (myproject)"
        direction TB
        ConfigFile["scrapy.cfg"]:::scrapy
        Settings["settings.py"]:::scrapy
        Items["items.py"]:::scrapy
        Pipelines["pipelines.py"]:::scrapy
        Spider["download_spider.py"]:::scrapy
    end

    %% File Store
    Downloads["downloads/"]:::datastore

    %% Connections
    EnvExample -->|provides template| Env
    Env -->|load variables| TestEnv
    Env -->|load variables| Sel
    TestEnv -->|validate env| Sel
    Sel -->|HTTP & Browser automation| JPopsuki
    Sel -->|writes links| Links
    Links -->|reads links| Spider
    Settings -->|configures| Spider
    Items -->|defines data model| Spider
    Pipelines -->|handles .torrent write| Spider
    Spider -->|downloads .torrent files| Downloads

    %% Click Events
    click EnvExample "https://github.com/umairahmedimran/script-jpop/blob/main/.env.example"
    click Env "https://github.com/umairahmedimran/script-jpop/blob/main/.env"
    click TestEnv "https://github.com/umairahmedimran/script-jpop/blob/main/test_env.py"
    click Sel "https://github.com/umairahmedimran/script-jpop/blob/main/sel.py"
    click ConfigFile "https://github.com/umairahmedimran/script-jpop/blob/main/myproject/scrapy.cfg"
    click Settings "https://github.com/umairahmedimran/script-jpop/blob/main/myproject/myproject/settings.py"
    click Items "https://github.com/umairahmedimran/script-jpop/blob/main/myproject/myproject/items.py"
    click Pipelines "https://github.com/umairahmedimran/script-jpop/blob/main/myproject/myproject/pipelines.py"
    click Spider "https://github.com/umairahmedimran/script-jpop/blob/main/myproject/myproject/spiders/download_spider.py"

    %% Styles
    classDef external fill:#d0e6f7,stroke:#6aaed6,color:#03396c,stroke-width:1px;
    classDef config fill:#e2f0d9,stroke:#8bbf6b,color:#33691e,stroke-width:1px;
    classDef scraper fill:#fde0dc,stroke:#f28b82,color:#990000,stroke-width:1px;
    classDef datastore fill:#fff4e6,stroke:#f9a825,color:#bf360c,stroke-width:1px;
    classDef scrapy fill:#e8eaf6,stroke:#5c6bc0,color:#1a237e,stroke-width:1px;
```
