# yuukiy/javsp architecture

> Architecture diagram of the GitHub repository yuukiy/javsp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuukiy/javsp
- Repository: https://github.com/yuukiy/javsp

## Overview

##### Project Type and Purpose
- Type: A command-line, full-pipeline metadata scraper & organizer (no WebUI).
- Purpose:
– Parse video file names for AV identifiers.
– Crawl multiple external AV metadata websites in parallel.
– Aggregate, normalize, translate, and crop cover images.
– Generate NFO metadata files & organize folders for media servers (Emby/Jellyfin/Kodi).
– Auto-update itself, configurable via YAML or CLI flags.

##### File Structure
- Top-level “javsp” package: core modules.
- Subpackages:
– web/… – one crawler module per site (e.g. javbus.py, fanza.py).
– cropper/… – image-cropping interfaces & implementations (slimeface, utils).
- Core modules:
– __main__.py (CLI entry)
– config.py (loads config.yml)
– file.py (filename parsing & file‐ops)
– datatype.py, nfo.py (data models & NFO generation)
– lib.py, func.py (orchestration, threading, aggregation)
– image.py (cover downloading)
– prompt.py/translate.py (AI vs translation)
– print.py (console feedback)
- Support directories: data/ (static lookup), tools/ (migration & helpers), unittest/ (test harness).
- Deployment: Docker/Dockerfile, GitHub Actions workflows.

##### README Insights
- Multithreaded crawling, daily tests (CI).
- Uses AI-based cropping, translation service hooks.
- YAML-based configuration + CLI override.
- No persistent database; file system and JSON/NFO outputs.

##### Main Components & Interactions

a. CLI Layer
- Entry: javsp/__main__.py
- Responsibilities: parse CLI args, load config, invoke orchestration.

b. Configuration Module
- config.py reads config.yml / envrc / CLI flags.
- Supplies settings to all modules.

c. Filename Parser & File Manager
- file.py extracts AV codes, handles file system scanning, folder creation.

d. Crawler Manager (Orchestrator)
- func.py / lib.py create thread pool.
- Distributes AV codes to multiple WebCrawler workers.

e. WebCrawler Workers
- javsp/web/*.py – one class per site implements HTTP requests, parsing, exception handling.
- To external web APIs/pages.
- Return normalized metadata objects.

f. Data Aggregation & Normalization
- datatype.py defines models (Title, Cast, Genres, etc.).
- lib.py merges data from multiple sources, resolves conflicts.

g. Translation Service
- prompt.py + web/translate.py call external translation API if enabled.

h. Image Downloader & Cropper
- image.py downloads high-res covers.
- cropper/interface.py selects correct algorithm.
- cropper/slimeface_crop.py applies AI cropping.

i. NFO Generator & File Writer
- nfo.py formats metadata into NFO/XML.
- print.py & file.py write NFO, artwork, organize directories.

j. Auxiliary Tools & Tests
- tools/* for config migration, version.
- unittest/* for unit and integration tests.

k. Deployment & CI
- Dockerfile → containerizes the entire app.
- GitHub Actions → runs unit tests, crawler tests, builds Docker image.

##### Architectural Patterns
- Pipeline / Orchestration: sequential stages (parse → crawl → aggregate → transform → write).
- Strategy Pattern: one crawler per site, interchangeable.
- Factory Pattern: instantiate crawler based on site code.
- Thread Pool Concurrency: concurrent web requests.
- Plugin-like extensibility: adding new crawler modules under web/.
- Configuration-driven vs code-driven behavior.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Pipeline
    subgraph "CLI Entry Point"
        style CLI fill:#cce5ff,stroke:#004085
        CLI["CLI Layer<br/>(javsp/__main__.py)"]:::internal
    end
    CLI -->|loads| ConfigFile
    CLI -->|invokes| FileMgr

    subgraph "Configuration"
        style ConfigFile fill:#fff3cd,stroke:#856404
        ConfigFile["config.yml"]:::io
        style ConfigMod fill:#cce5ff,stroke:#004085
        ConfigMod["Configuration Module<br/>(javsp/config.py)"]:::internal
    end
    ConfigFile --> ConfigMod

    subgraph "Filename Parsing & File Management"
        style FileMgr fill:#cce5ff,stroke:#004085
        FileMgr["File Manager<br/>(javsp/file.py)"]:::internal
    end

    CLI --> FileMgr

    subgraph "Orchestration"
        style Orch1 fill:#cce5ff,stroke:#004085
        Orch1["Crawler Manager<br/>(javsp/func.py)"]:::internal
        style Orch2 fill:#cce5ff,stroke:#004085
        Orch2["Thread Pool & Merge<br/>(javsp/lib.py)"]:::internal
    end
    FileMgr --> Orch1
    ConfigMod --> Orch1
    Orch1 --> Orch2

    subgraph "WebCrawler Workers (Plugins)"
        style Base fill:#cce5ff,stroke:#004085
        Base["Crawler Base<br/>(javsp/web/base.py)"]:::internal
        style Excp fill:#cce5ff,stroke:#004085
        Excp["Exceptions<br/>(javsp/web/exceptions.py)"]:::internal
        style Javbus fill:#cce5ff,stroke:#004085
        Javbus["javbus Crawler<br/>(javsp/web/javbus.py)"]:::internal
        style Fanza fill:#cce5ff,stroke:#004085
        Fanza["fanza Crawler<br/>(javsp/web/fanza.py)"]:::internal
        style Avsox fill:#cce5ff,stroke:#004085
        Avsox["avsox Crawler<br/>(javsp/web/avsox.py)"]:::internal
    end
    Orch2 -->|dispatch| Base
    Base --> Javbus
    Base --> Fanza
    Base --> Avsox
    Base --> Excp
    subgraph "External AV Sites"
        style ExternalWeb fill:#d4edda,stroke:#155724
        ExternalWeb["AV Metadata Websites"]:::external
    end
    Javbus --> ExternalWeb
    Fanza --> ExternalWeb
    Avsox --> ExternalWeb

    subgraph "Data Models & Aggregation"
        style Models fill:#cce5ff,stroke:#004085
        Models["Data Models<br/>(javsp/datatype.py)"]:::internal
        style MergeLogic fill:#cce5ff,stroke:#004085
        MergeLogic["Aggregation Logic<br/>(javsp/lib.py)"]:::internal
    end
    ExternalWeb --> Models
    Models --> MergeLogic
    Orch2 --> MergeLogic

    subgraph "Translation Service"
        style Prompt fill:#cce5ff,stroke:#004085
        Prompt["Prompt Module<br/>(javsp/prompt.py)"]:::internal
        style Trans fill:#cce5ff,stroke:#004085
        Trans["Translator<br/>(javsp/web/translate.py)"]:::internal
        style TranslateAPI fill:#d4edda,stroke:#155724
        TranslateAPI["External Translation API"]:::external
    end
    MergeLogic --> Prompt
    Prompt --> Trans
    Trans --> TranslateAPI

    subgraph "Image Download & Cropping"
        style ImgDL fill:#cce5ff,stroke:#004085
        ImgDL["Image Downloader<br/>(javsp/image.py)"]:::internal
        style CropInt fill:#cce5ff,stroke:#004085
        CropInt["Crop Interface<br/>(javsp/cropper/interface.py)"]:::internal
        style Slime fill:#cce5ff,stroke:#004085
        Slime["Slimeface Cropper<br/>(javsp/cropper/slimeface_crop.py)"]:::internal
        style Utils fill:#cce5ff,stroke:#004085
        Utils["Cropper Utils<br/>(javsp/cropper/utils.py)"]:::internal
    end
    MergeLogic --> ImgDL
    ImgDL --> CropInt
    CropInt --> Slime
    CropInt --> Utils

    subgraph "NFO Generation & Writing"
        style NFO fill:#cce5ff,stroke:#004085
        NFO["NFO Generator<br/>(javsp/nfo.py)"]:::internal
        style PrintMod fill:#cce5ff,stroke:#004085
        PrintMod["Console Printer<br/>(javsp/print.py)"]:::internal
        style FSWriter fill:#cce5ff,stroke:#004085
        FSWriter["File Writer<br/>(javsp/file.py)"]:::internal
        style OutputStore fill:#fff3cd,stroke:#856404
        OutputStore["Output Folders & NFOs"]:::io
    end
    Slime --> NFO
    Utils --> NFO
    NFO --> PrintMod
    PrintMod --> FSWriter
    FSWriter --> OutputStore

    subgraph "Auxiliary Tools & Tests"
        style Tools1 fill:#fff3cd,stroke:#856404
        Tools1["tools/config_migration.py"]:::io
        style Tools2 fill:#fff3cd,stroke:#856404
        Tools2["tools/version.py"]:::io
        style Tests fill:#fff3cd,stroke:#856404
        Tests["unittest/"]:::io
    end

    subgraph "Deployment & CI"
        style Dockerf fill:#fff3cd,stroke:#856404
        Dockerf["docker/Dockerfile"]:::io
        style CI1 fill:#fff3cd,stroke:#856404
        CI1[".github/workflows/cx_freeze.yml"]:::io
        style CI2 fill:#fff3cd,stroke:#856404
        CI2[".github/workflows/docker-build.yml"]:::io
        style CI3 fill:#fff3cd,stroke:#856404
        CI3[".github/workflows/test-basic-funcs.yml"]:::io
        style CI4 fill:#fff3cd,stroke:#856404
        CI4[".github/workflows/test-web-funcs.yml"]:::io
    end

    %% Click Events
    click CLI "https://github.com/yuukiy/javsp/blob/master/javsp/__main__.py"
    click ConfigMod "https://github.com/yuukiy/javsp/blob/master/javsp/config.py"
    click ConfigFile "https://github.com/yuukiy/javsp/blob/master/config.yml"
    click FileMgr "https://github.com/yuukiy/javsp/blob/master/javsp/file.py"
    click Orch1 "https://github.com/yuukiy/javsp/blob/master/javsp/func.py"
    click Orch2 "https://github.com/yuukiy/javsp/blob/master/javsp/lib.py"
    click Base "https://github.com/yuukiy/javsp/blob/master/javsp/web/base.py"
    click Excp "https://github.com/yuukiy/javsp/blob/master/javsp/web/exceptions.py"
    click Javbus "https://github.com/yuukiy/javsp/blob/master/javsp/web/javbus.py"
    click Fanza "https://github.com/yuukiy/javsp/blob/master/javsp/web/fanza.py"
    click Avsox "https://github.com/yuukiy/javsp/blob/master/javsp/web/avsox.py"
    click Models "https://github.com/yuukiy/javsp/blob/master/javsp/datatype.py"
    click MergeLogic "https://github.com/yuukiy/javsp/blob/master/javsp/lib.py"
    click Prompt "https://github.com/yuukiy/javsp/blob/master/javsp/prompt.py"
    click Trans "https://github.com/yuukiy/javsp/blob/master/javsp/web/translate.py"
    click ImgDL "https://github.com/yuukiy/javsp/blob/master/javsp/image.py"
    click CropInt "https://github.com/yuukiy/javsp/blob/master/javsp/cropper/interface.py"
    click Slime "https://github.com/yuukiy/javsp/blob/master/javsp/cropper/slimeface_crop.py"
    click Utils "https://github.com/yuukiy/javsp/blob/master/javsp/cropper/utils.py"
    click NFO "https://github.com/yuukiy/javsp/blob/master/javsp/nfo.py"
    click PrintMod "https://github.com/yuukiy/javsp/blob/master/javsp/print.py"
    click Tools1 "https://github.com/yuukiy/javsp/blob/master/tools/config_migration.py"
    click Tools2 "https://github.com/yuukiy/javsp/blob/master/tools/version.py"
    click Tests "https://github.com/yuukiy/javsp/tree/master/unittest/"
    click Dockerf "https://github.com/yuukiy/javsp/tree/master/docker/Dockerfile"
    click CI1 "https://github.com/yuukiy/javsp/blob/master/.github/workflows/cx_freeze.yml"
    click CI2 "https://github.com/yuukiy/javsp/blob/master/.github/workflows/docker-build.yml"
    click CI3 "https://github.com/yuukiy/javsp/blob/master/.github/workflows/test-basic-funcs.yml"
    click CI4 "https://github.com/yuukiy/javsp/blob/master/.github/workflows/test-web-funcs.yml"

    %% Styles
    classDef internal fill:#cce5ff,stroke:#004085,color:#004085
    classDef external fill:#d4edda,stroke:#155724,color:#155724
    classDef io fill:#fff3cd,stroke:#856404,color:#856404
```
