# antiheroguy/one-piece-image-crawler architecture

> Architecture diagram of the GitHub repository antiheroguy/one-piece-image-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antiheroguy/one-piece-image-crawler
- Repository: https://github.com/antiheroguy/one-piece-image-crawler

## Overview

##### Main Components
- Node.js Runtime Environment – All the scripts (crawling and downloading) are executed using Node.
- Configuration Module – The “config” directory (with config/config.json) holds configuration parameters that the scripts can use.
- Database Layer – Represented by:
– Migrations (the “migrations” directory, e.g., 20230825044228-create-item.js) that set up the underlying schema.
– Models (in “models” like index.js and item.js) that define database tables and relationships. This is where crawled data is stored.
- Logging – A “logs” directory to capture runtime logs.
- Crawler Module – opts-crawl.js is tasked with crawling data from external sources and saving image metadata (or similar details) to the database.
- Download Modules – There are two download scripts:
– opts-download.js (for the “Thousand Storm” mode) which downloads images based on data (likely from the database or external sources).
– optc-download.js (for the “Treasure Cruise” mode) which takes parameters (like a save path and the number of images) and downloads images accordingly.

##### Relationships and Data Flow
- From the configuration module, the application reads necessary environmental and system parameters.
- When setting up the project, “yarn migrate” is run to initialize the database structure using the migration scripts.
- The download modules then read either from the database or interact with external image sources:
– For “Thousand Storm”, opts-download.js downloads images
– For “Treasure Cruise”, optc-download.js initiates the download process based on command-line parameters.
- Both download processes may interact with the file system (to save images) and log their activities in the logs directory.

##### Architectural Patterns and Technologies
- This project follows a modular structure separating concerns (configuration, data persistence, crawling logic, and image downloading logic).
- The use of migrations and models suggests an ORM or a similar abstraction for managing data, even though the specific database engine is not detailed.
- The scripts are designed to be run as discrete CLI processes which interact with one another indirectly via the shared database and configuration.
- The clear separation between crawling (data ingestion) and downloading (data consumption) is a key architectural pattern here.

## Diagram (Mermaid)

```mermaid
flowchart TD
    R["Node.js Runtime Environment"]:::runtime
    click R "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/package.json"
    
    C["Configuration Module"]:::config
    click C "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/config/config.json"
    
    R --> C
    
    subgraph "Database Layer"
        MIG["Migrations (Schema Setup)"]:::db
        click MIG "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/migrations/20230825044228-create-item.js"
        MODE1["Models (index.js)"]:::db
        click MODE1 "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/models/index.js"
        MODE2["Models (item.js)"]:::db
        click MODE2 "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/models/item.js"
    end
    
    CW["Crawler Module"]:::process
    click CW "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/opts-crawl.js"
    
    DT["Download Module - Thousand Storm"]:::process
    click DT "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/opts-download.js"
    
    DC["Download Module - Treasure Cruise"]:::process
    click DC "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/optc-download.js"
    
    L["Logging"]:::log
    click L "https://github.com/antiheroguy/one-piece-image-crawler/blob/master/logs/.gitkeep"
    
    C --> CW
    C --> DT
    C --> DC
    C --> MIG
    
    CW -->|"saves data"| MODE1
    CW -->|"saves data"| MODE2
    
    MODE1 -.->|"optional data"| DT
    MODE2 -.->|"optional data"| DC
    
    DT -->|"logs activity"| L
    DC -->|"logs activity"| L

    classDef runtime fill:#f9c,stroke:#333,stroke-width:2px;
    classDef config fill:#ccf,stroke:#333,stroke-width:2px;
    classDef db fill:#cfc,stroke:#333,stroke-width:2px;
    classDef process fill:#ffc,stroke:#333,stroke-width:2px;
    classDef log fill:#fcc,stroke:#333,stroke-width:2px;
```
