How yuukiy/javsp works
Project Type and Purpose
• Type: A command-line, full-pipeline metadata scraper & organizer (no WebUI).
• Purpose:
– Parse video file names for AV identifiers.
– Crawl multiple external AV metadata websites in parallel.
– Aggregate, normalize, translate, and crop cover images.
– Generate NFO metadata files & organize folders for media servers (Emby/Jellyfin/Kodi).
– Auto-update itself, configurable via YAML or CLI flags.
File Structure
• Top-level “javsp” package: core modules.
• Subpackages:
– web/… – one crawler module per site (e.g. javbus.py, fanza.py).
– cropper/… – image-cropping interfaces & implementations (slimeface, utils).
• Core modules:
– __main__.py (CLI entry)
– config.py (loads config.yml)
– file.py (filename parsing & file‐ops)
– datatype.py, nfo.py (data models & NFO generation)
– lib.py, func.py (orchestration, threading, aggregation)
– image.py (cover downloading)
– prompt.py/translate.py (AI vs translation)
– print.py (console feedback)
• Support directories: data/ (static lookup), tools/ (migration & helpers), unittest/ (test harness).
• Deployment: Docker/Dockerfile, GitHub Actions workflows.
README Insights
• Multithreaded crawling, daily tests (CI).
• Uses AI-based cropping, translation service hooks.
• YAML-based configuration + CLI override.
• No persistent database; file system and JSON/NFO outputs.
Main Components & Interactions
a. CLI Layer
• Entry: javsp/__main__.py
• Responsibilities: parse CLI args, load config, invoke orchestration.
b. Configuration Module
• config.py reads config.yml / envrc / CLI flags.
• Supplies settings to all modules.
c. Filename Parser & File Manager
• file.py extracts AV codes, handles file system scanning, folder creation.
d. Crawler Manager (Orchestrator)
• func.py / lib.py create thread pool.
• Distributes AV codes to multiple WebCrawler workers.
e. WebCrawler Workers
• javsp/web/*.py – one class per site implements HTTP requests, parsing, exception handling.
• To external web APIs/pages.
• Return normalized metadata objects.
f. Data Aggregation & Normalization
• datatype.py defines models (Title, Cast, Genres, etc.).
• lib.py merges data from multiple sources, resolves conflicts.
g. Translation Service
• prompt.py + web/translate.py call external translation API if enabled.
h. Image Downloader & Cropper
• image.py downloads high-res covers.
• cropper/interface.py selects correct algorithm.
• cropper/slimeface_crop.py applies AI cropping.
i. NFO Generator & File Writer
• nfo.py formats metadata into NFO/XML.
• print.py & file.py write NFO, artwork, organize directories.
j. Auxiliary Tools & Tests
• tools/* for config migration, version.
• unittest/* for unit and integration tests.
k. Deployment & CI
• Dockerfile → containerizes the entire app.
• GitHub Actions → runs unit tests, crawler tests, builds Docker image.
Architectural Patterns
• Pipeline / Orchestration: sequential stages (parse → crawl → aggregate → transform → write).
• Strategy Pattern: one crawler per site, interchangeable.
• Factory Pattern: instantiate crawler based on site code.
• Thread Pool Concurrency: concurrent web requests.
• Plugin-like extensibility: adding new crawler modules under web/.
• Configuration-driven vs code-driven behavior.