How damian666/literotica-dataset-creator works
Project Type and Purpose
• This is a Python-based ETL/scraping pipeline.
• It crawls literotica.com in stages (categories → page counts → story links → downloads → post-processing).
• Final outputs include collated stories, enriched metadata, cleaned datasets, and samples.
Main Components
a. Configuration Layer
– config.json (plus config.example.json)
– Drives endpoints, file‐paths, rate limits
b. Scraping/Extraction Stages (data_P0 … data_P5)
– P0: get_categories
– P1: get_page_counts
– P2: get_story_links
– P3: download_stories
– P4: download_extra_pages
– P5: collate_stories
c. Post-Processing Modules
– enrich_metadata.py
– remove_duplicates.py
– remove_faulty_entries.py
– validate.py & validate_meta_structure.py
– sample_dataset.py
d. Shared Library
– utils.py (HTTP wrappers, parsers, I/O helpers)
e. External Service
– literotica.com (HTTP endpoints)
f. Persistence
– Local file system directories for categories, listings, raw HTML, JSON outputs, final dataset
Relationships and Data Flow
• config.json → all scripts (injects URLs, output dirs)
• data_P0 → reads config → fetches category URLs → writes categories.txt/JSON
• data_P1 → reads categories → fetches page counts → writes page_counts/
• data_P2 → reads page_counts → extracts story links → writes links/
• data_P3 → reads links → downloads story HTML → writes stories/
• data_P4 → reads stories → grabs extra pages (images, author pages) → writes extra_pages/
• data_P5 → reads all raw pages → collates into unified JSON/CSV → writes collated/
• enrich_metadata → reads collated → adds tags/dates/length → writes enriched/
• remove_duplicates & remove_faulty_entries → read enriched → filter bad entries → write cleaned/
• validate & validate_meta_structure → run checks on cleaned → report errors
• sample_dataset → reads cleaned → produces sample subset for analysis
Architectural Patterns and Design Principles
• Pipeline (linear DAG) – each stage depends on the prior stage’s output
• Modular single‐responsibility scripts
• Configuration‐driven behavior
• Shared utility library for DRY code
• File‐based staging between stages (loose coupling)
Technologies and Frameworks
• Language: Python 3.x
• HTTP: requests
• HTML parsing: beautifulsoup4
• JSON/CSV I/O: built-in libraries
• Local FS (no external database)