How wenshihken/marumaru-x-crawler works
Project type and purpose
• This is an open-source data-crawling tool (not a web or full-stack app).
• Its sole purpose is to fetch song lists and lyrics from the marumaru-x website and store them as JSON files.
File structure
• Top-level Python scripts:
• song_list_crawler.py – fetches the list of songs (title, thumbnail URL, duration, link)
• lyrics_crawler.py – consumes the song list, navigates to each song page, and extracts detailed lyrics
• config.py – holds shared configuration (e.g., base URL, concurrency settings, output paths)
• requirements.txt – external dependencies (Playwright, any HTTP or JSON libraries)
• LYRICS_DIR/ – output directory for crawled JSON files
• LICENSE, readme.md – meta files
Key insights from README
• Dependencies: Python packages and Playwright browser driver
• Usage: two-step crawling (first song list, then lyrics)
• Output: well-defined JSON schemas for both song list and lyrics details
Main components
a. External Service
• marumaru-x website (HTTP endpoints for song list and individual lyrics pages)
b. Crawlers
• SongListCrawler (song_list_crawler.py)
• LyricsCrawler (lyrics_crawler.py)
c. Shared Configuration
• Config module (config.py)
d. Browser Automation Engine
• Playwright (underlying headless browser or driver)
e. Local Storage
• LYRICS_DIR filesystem (JSON output)
f. Dependencies
• Python runtime, third-party libraries (specified in requirements.txt)
Relationships and data flows
• SongListCrawler reads config → issues HTTP requests to marumaru-x → parses HTML/JSON → writes song_list.json (intermediate) → Signals LyricsCrawler
• LyricsCrawler reads config + song_list.json → for each entry launches Playwright browser session → navigates to the song’s lyrics page → extracts metadata + timed lyrics → writes individual JSON files in LYRICS_DIR
• Both crawlers depend on requirements.txt modules; config.py is imported by both
Architectural patterns and design principles
• Single-responsibility: two distinct crawler modules
• Config centralization: all environment/URL settings in config.py
• Pipeline flow: list crawler → lyrics crawler
• File-based hand-off: intermediate JSON as a simple queue
Technologies / frameworks
• Python 3.x
• Playwright for browser automation
• Standard JSON library for serialization
• Any HTTP/HTML parser (e.g., requests, BeautifulSoup, or Playwright’s page.evaluate)
Diagramming recommendations
• Blue rectangles for Python modules (SongListCrawler, LyricsCrawler, Config)
• From config.py → both crawlers (import/dependency)
• From SongListCrawler → marumaru-x → SongListCrawler (request/response)
• From SongListCrawler → JSON output → LyricsCrawler (data hand-off)
• From LyricsCrawler → marumaru-x via Playwright → JSON files