# wenshihken/marumaru-x-crawler architecture

> Architecture diagram of the GitHub repository wenshihken/marumaru-x-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wenshihken/marumaru-x-crawler
- Repository: https://github.com/wenshihken/marumaru-x-crawler

## Overview

##### Project type and purpose
- This is an open-source data-crawling tool (not a web or full-stack app).
- Its sole purpose is to fetch song lists and lyrics from the marumaru-x website and store them as JSON files.

##### File structure
- Top-level Python scripts:
- song_list_crawler.py – fetches the list of songs (title, thumbnail URL, duration, link)
- lyrics_crawler.py – consumes the song list, navigates to each song page, and extracts detailed lyrics
- config.py – holds shared configuration (e.g., base URL, concurrency settings, output paths)
- requirements.txt – external dependencies (Playwright, any HTTP or JSON libraries)
- LYRICS_DIR/ – output directory for crawled JSON files
- LICENSE, readme.md – meta files

##### Key insights from README
- Dependencies: Python packages and Playwright browser driver
- Usage: two-step crawling (first song list, then lyrics)
- Output: well-defined JSON schemas for both song list and lyrics details

##### Main components
a. External Service
- marumaru-x website (HTTP endpoints for song list and individual lyrics pages)
b. Crawlers
- SongListCrawler (song_list_crawler.py)
- LyricsCrawler (lyrics_crawler.py)
c. Shared Configuration
- Config module (config.py)
d. Browser Automation Engine
- Playwright (underlying headless browser or driver)
e. Local Storage
- LYRICS_DIR filesystem (JSON output)
f. Dependencies
- Python runtime, third-party libraries (specified in requirements.txt)

##### Relationships and data flows
- SongListCrawler reads config → issues HTTP requests to marumaru-x → parses HTML/JSON → writes song_list.json (intermediate) → Signals LyricsCrawler
- LyricsCrawler reads config + song_list.json → for each entry launches Playwright browser session → navigates to the song’s lyrics page → extracts metadata + timed lyrics → writes individual JSON files in LYRICS_DIR
- Both crawlers depend on requirements.txt modules; config.py is imported by both

##### Architectural patterns and design principles
- Single-responsibility: two distinct crawler modules
- Config centralization: all environment/URL settings in config.py
- Pipeline flow: list crawler → lyrics crawler
- File-based hand-off: intermediate JSON as a simple queue

##### Technologies / frameworks
- Python 3.x
- Playwright for browser automation
- Standard JSON library for serialization
- Any HTTP/HTML parser (e.g., requests, BeautifulSoup, or Playwright’s page.evaluate)

##### Diagramming recommendations
- Blue rectangles for Python modules (SongListCrawler, LyricsCrawler, Config)
- From config.py → both crawlers (import/dependency)
- From SongListCrawler → marumaru-x → SongListCrawler (request/response)
- From SongListCrawler → JSON output → LyricsCrawler (data hand-off)
- From LyricsCrawler → marumaru-x via Playwright → JSON files

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Service
    External["marumaru-x Service"]:::external

    %% Engine
    Engine["Playwright Engine"]:::engine

    %% Python Modules
    Config["Config (config.py)"]:::pythonModule
    SongListCrawler["SongListCrawler\n(song_list_crawler.py)"]:::pythonModule
    LyricsCrawler["LyricsCrawler\n(lyrics_crawler.py)"]:::pythonModule
    Dependencies["Dependencies\n(requirements.txt)"]:::pythonModule

    %% Storage
    Storage["LYRICS_DIR\n(JSON Output)"]:::storage

    %% Data Flow
    Config -->|import| SongListCrawler
    Config -->|import| LyricsCrawler
    Dependencies -->|provides libraries| SongListCrawler
    Dependencies -->|provides libraries| LyricsCrawler

    SongListCrawler -->|HTTP GET /songList| External
    External -->|200 OK HTML/JSON| SongListCrawler
    SongListCrawler -->|parse HTML| SongListCrawler
    SongListCrawler -->|write song_list.json| Storage
    Storage -->|read song_list.json| LyricsCrawler

    LyricsCrawler -->|launch browser| Engine
    Engine -->|navigate & fetch| External
    External -->|HTML page| Engine
    Engine -->|return data| LyricsCrawler
    LyricsCrawler -->|extract lyrics| LyricsCrawler
    LyricsCrawler -->|write JSON files| Storage

    %% Click Events
    click SongListCrawler "https://github.com/wenshihken/marumaru-x-crawler/blob/main/song_list_crawler.py"
    click LyricsCrawler "https://github.com/wenshihken/marumaru-x-crawler/blob/main/lyrics_crawler.py"
    click Config "https://github.com/wenshihken/marumaru-x-crawler/blob/main/config.py"
    click Dependencies "https://github.com/wenshihken/marumaru-x-crawler/blob/main/requirements.txt"
    click Storage "https://github.com/wenshihken/marumaru-x-crawler/tree/main/LYRICS_DIR/"

    %% Styles
    classDef pythonModule fill:#E1F5FE,stroke:#0288D1,stroke-width:1px
    classDef external fill:#C8E6C9,stroke:#2E7D32,stroke-width:1px
    classDef storage fill:#FFE0B2,stroke:#EF6C00,stroke-width:1px
    classDef engine fill:#CFD8DC,stroke:#546E7A,stroke-width:1px
```
