How kokiebisu/saola works
1. Purpose and overall shape
- This repository is a small Python-based scraping pipeline for manga on mangareader.to.
- The end-to-end flow is: take a manga URL, extract metadata and chapter listings, download chapter images into a local folder, then optionally compile those images into a single PDF.
2. Top-level entrypoints
- saola.py is the main orchestration script: it appears to combine scraping and PDF compilation in one run.
- scrape.py is the download-only entrypoint: it fetches manga content into a folder.
- compile.py is the post-processing entrypoint: it takes already-downloaded manga images and turns them into a PDF.
3. Core pipeline boundary
- The repo is structured around a clear separation between collection and compilation.
- Collection interacts with the website and filesystem to build a local chapter/image tree.
- Compilation operates on that local tree and does not depend on the remote site.
4. Scraping and extraction layer
- lib/extractor.py is likely the central domain layer for parsing manga pages, finding chapters, and extracting metadata like title and cover image.
- lib/beautifulsoup.py suggests HTML parsing is done with BeautifulSoup for static page content.
- lib/selenium.py indicates browser automation is used where the site requires dynamic rendering or JS-driven navigation.
5. Image acquisition layer
- lib/image.py is the image-handling subsystem: downloading chapter pages/images and preparing them for storage and later PDF generation.
- This layer likely turns the remote image URLs into ordered local files, preserving chapter/page sequence.
6. Shared utilities
- lib/utils.py is the cross-cutting support module for filesystem paths, naming, threading helpers, configuration loading, and general convenience functions.
- The README’s mention of settings.json and desktop folder customization implies utilities also manage user-specific paths and output layout.
7. Filesystem-centric data flow
- The primary data store is the local filesystem, not a database.
- The application creates a manga-named directory on the user’s desktop, stores chapter folders/images there, and writes the final PDF into that same structure.
- This makes the output folder both the intermediate artifact store and the final deliverable location.
8. Concurrency model
- The CLI supports a thread count option, so chapter/image downloading is parallelized.
- Concurrency is likely confined to the scraping/download stage, while PDF compilation is probably sequential because image order matters.
9. Configuration and user inputs
- The runtime is configured via command-line flags: URL, thread count, start/end chapter range, and language.
- The README also mentions optional settings.json, which suggests some behavior can be overridden without changing code, especially paths or default storage locations.
10. Runtime and packaging
- The project is Python-based and appears designed to run in containers rather than directly on the host.
- Dockerfile, docker-compose.yml, and .devcontainer/ indicate a reproducible development/runtime environment, likely to ensure Selenium/browser dependencies and Python packages are available consistently.
11. External dependency
- The only clearly identified external system is mangareader.to, which is the source of truth for manga metadata, chapter structure, and page images.
- The repo is therefore an ingestion tool that depends heavily on the site’s HTML structure and download endpoints.
12. Likely architecture diagram boundaries
- A good diagram would show:
- User/CLI input
- Orchestrator scripts (saola.py, scrape.py, compile.py)
- Scraping/extraction modules (beautifulsoup, selenium, extractor)
- Image/file handling (image, utils)
- Local output folder on desktop
- Optional PDF compilation step
- External source website (mangareader.to)
- The key architectural constraint is that remote fetching and local compilation are separate phases connected by filesystem artifacts rather than by in-memory objects or a service API.
Main entry points: extractor.py Extractor · image.py Image · saola.py saola.py · scrape.py scrape.py
Components 17
Entrypoints
CLI scripts that start scraping or compilation runs.
- saola.py · Main end-to-end entrypoint that combines scraping and PDF compilation.
saola.py - scrape.py · Download-only entrypoint for fetching manga into local folders.
scrape.py - compile.py · Post-processing entrypoint that turns downloaded images into a PDF.
compile.py - User input · CLI parameters such as URL, thread count, chapter range, and language.
Pipeline
Core scraping, download, and compile subsystems.
- Extractor · Domain layer for manga metadata, chapter discovery, and page extraction.
lib/extractor.py - BeautifulSoup · Parses static HTML content from manga pages.
lib/beautifulsoup.py - Selenium · Handles dynamic pages and JS-driven navigation when static parsing is not enough.
lib/selenium.py - Image · Downloads chapter images and prepares them for local storage and PDF input.
lib/image.py - Utils · Cross-cutting helpers for paths, naming, threading, and runtime configuration.
lib/utils.py - Output tree · Local manga folder tree on disk used for intermediate chapters, images, and final PDF.
- PDF output · Final PDF generated from the downloaded image set.
- mangareader.to · External source of manga metadata, chapters, and image assets.
- settings.json · Optional local configuration for defaults and output locations.
Runtime
Container and environment files used to run the tool consistently.
- Dockerfile · Container image definition for reproducible runtime setup.
Dockerfile - docker-compose.yml · Local container orchestration for the application stack.
docker-compose.yml - .devcontainer · Development container setup for consistent Python and browser dependencies.
.devcontainer/devcontainer.json - requirements · Python package dependencies for scraping and compilation.
requirements.txt
Connections 22
- User input → saola.py: invoke · User launches the combined pipeline through the main script.
- User input → scrape.py: invoke · User can launch the download-only workflow.
- User input → compile.py: invoke · User can launch the PDF-only workflow.
- saola.py → Extractor: extract · Main run parses metadata and chapter structure.
- saola.py → Image: download · Main run downloads chapter images.
- saola.py → PDF output: compile · Main run can produce a PDF after scraping.
- scrape.py → Extractor: extract · Scrape workflow resolves manga and chapter information.
- scrape.py → Image: download · Scrape workflow stores chapter images locally.
- compile.py → PDF output: build · Compilation workflow assembles a PDF from local images.
- Extractor → BeautifulSoup: parse html · Static parsing is used for page and metadata extraction.
- Extractor → Selenium: fallback · Dynamic browser automation is used when needed.
- BeautifulSoup → mangareader.to: read pages · HTML parser consumes pages from the source site.
- Selenium → mangareader.to: browse · Browser automation interacts with the source site.
- Image → mangareader.to: fetch images · Image subsystem pulls page images from remote URLs.
- Image → Utils: paths · Uses shared helpers for naming, ordering, and filesystem layout.
- Extractor → Utils: helpers · Uses shared helpers for paths, config, and runtime options.
- Image → Output tree: write files · Downloads are persisted into the local manga folder tree.
- PDF output → Output tree: read input · Compiler reads the local image tree and writes the final PDF there.
- settings.json → Utils: configure · Optional settings influence default paths and runtime behavior.
- Dockerfile → requirements: installs · Container build uses Python dependencies.
- docker-compose.yml → Dockerfile: runs · Compose starts the built application container.
- .devcontainer → Dockerfile: extends · Development container builds on the same runtime image.