# damian666/literotica-dataset-creator architecture

> Architecture diagram of the GitHub repository damian666/literotica-dataset-creator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damian666/literotica-dataset-creator
- Repository: https://github.com/damian666/literotica-dataset-creator

## Overview

##### Project Type and Purpose
- This is a Python-based ETL/scraping pipeline.
- It crawls literotica.com in stages (categories → page counts → story links → downloads → post-processing).
- Final outputs include collated stories, enriched metadata, cleaned datasets, and samples.

##### Main Components
a. Configuration Layer
– config.json (plus config.example.json)
– Drives endpoints, file‐paths, rate limits
b. Scraping/Extraction Stages (data_P0 … data_P5)
– P0: get_categories
– P1: get_page_counts
– P2: get_story_links
– P3: download_stories
– P4: download_extra_pages
– P5: collate_stories
c. Post-Processing Modules
– enrich_metadata.py
– remove_duplicates.py
– remove_faulty_entries.py
– validate.py & validate_meta_structure.py
– sample_dataset.py
d. Shared Library
– utils.py (HTTP wrappers, parsers, I/O helpers)
e. External Service
– literotica.com (HTTP endpoints)
f. Persistence
– Local file system directories for categories, listings, raw HTML, JSON outputs, final dataset

##### Relationships and Data Flow
- config.json → all scripts (injects URLs, output dirs)
- data_P0 → reads config → fetches category URLs → writes categories.txt/JSON
- data_P1 → reads categories → fetches page counts → writes page_counts/
- data_P2 → reads page_counts → extracts story links → writes links/
- data_P3 → reads links → downloads story HTML → writes stories/
- data_P4 → reads stories → grabs extra pages (images, author pages) → writes extra_pages/
- data_P5 → reads all raw pages → collates into unified JSON/CSV → writes collated/
- enrich_metadata → reads collated → adds tags/dates/length → writes enriched/
- remove_duplicates & remove_faulty_entries → read enriched → filter bad entries → write cleaned/
- validate & validate_meta_structure → run checks on cleaned → report errors
- sample_dataset → reads cleaned → produces sample subset for analysis

##### Architectural Patterns and Design Principles
- Pipeline (linear DAG) – each stage depends on the prior stage’s output
- Modular single‐responsibility scripts
- Configuration‐driven behavior
- Shared utility library for DRY code
- File‐based staging between stages (loose coupling)

##### Technologies and Frameworks
- Language: Python 3.x
- HTTP: requests
- HTML parsing: beautifulsoup4
- JSON/CSV I/O: built-in libraries
- Local FS (no external database)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Utilities
    config["config.json"]:::config
    utils["utils.py"]:::config

    %% External Service
    ext["literotica.com"]:::external

    %% Extraction Layer
    subgraph "Extraction"
        script0["data_P0_get_categories.py"]:::extraction
        catDir["categories/"]:::storage
        script1["data_P1_get_page_counts.py"]:::extraction
        pcDir["page_counts/"]:::storage
        script2["data_P2_get_story_links.py"]:::extraction
        linksDir["links/"]:::storage
        script3["data_P3_download_stories.py"]:::extraction
        storiesDir["stories/"]:::storage
        script4["data_P4_download_extra_pages.py"]:::extraction
        extraDir["extra_pages/"]:::storage
        script5["data_P5_collate_stories.py"]:::extraction
        collatedDir["collated/"]:::storage
    end

    %% Post-Processing Layer
    subgraph "Post-Processing"
        enrich["enrich_metadata.py"]:::post
        enrichedDir["enriched/"]:::storage
        dup["remove_duplicates.py"]:::post
        faulty["remove_faulty_entries.py"]:::post
        cleanedDir["cleaned/"]:::storage
        val1["validate.py"]:::post
        val2["validate_meta_structure.py"]:::post
        sample["sample_dataset.py"]:::post
        sampleOut["samples/"]:::storage
    end

    %% Data Flow Extraction
    config -->|loads config| script0
    config -->|loads config| script1
    config -->|loads config| script2
    config -->|loads config| script3
    config -->|loads config| script4
    config -->|loads config| script5

    utils --> script0
    utils --> script1
    utils --> script2
    utils --> script3
    utils --> script4
    utils --> script5

    script0 -->|HTTP request| ext
    script1 -->|HTTP request| ext
    script2 -->|HTTP request| ext
    script3 -->|HTTP request| ext
    script4 -->|HTTP request| ext

    script0 -->|"categories list"| catDir
    catDir --> script1
    script1 -->|"page counts"| pcDir
    pcDir --> script2
    script2 -->|"story links"| linksDir
    linksDir --> script3
    script3 -->|"HTML pages"| storiesDir
    storiesDir --> script4
    script4 -->|"extra pages"| extraDir
    extraDir --> script5
    script5 -->|"collated data"| collatedDir

    %% Data Flow Post-Processing
    config -->|loads config| enrich
    config -->|loads config| dup
    config -->|loads config| faulty
    config -->|loads config| val1
    config -->|loads config| val2
    config -->|loads config| sample

    utils --> enrich
    utils --> dup
    utils --> faulty
    utils --> val1
    utils --> val2
    utils --> sample

    collatedDir --> enrich
    enrich -->|"enriched data"| enrichedDir
    enrichedDir --> dup
    enrichedDir --> faulty
    dup -->|"cleaned data"| cleanedDir
    faulty -->|"cleaned data"| cleanedDir
    cleanedDir --> val1
    cleanedDir --> val2
    cleanedDir --> sample
    sample -->|"sample subset"| sampleOut

    %% Click Events
    click config "https://github.com/damian666/literotica-dataset-creator/blob/master/config.json"
    click utils "https://github.com/damian666/literotica-dataset-creator/blob/master/utils.py"
    click script0 "https://github.com/damian666/literotica-dataset-creator/blob/master/data_P0_get_categories.py"
    click script1 "https://github.com/damian666/literotica-dataset-creator/blob/master/data_P1_get_page_counts.py"
    click script2 "https://github.com/damian666/literotica-dataset-creator/blob/master/data_P2_get_story_links.py"
    click script3 "https://github.com/damian666/literotica-dataset-creator/blob/master/data_P3_download_stories.py"
    click script4 "https://github.com/damian666/literotica-dataset-creator/blob/master/data_P4_download_extra_pages.py"
    click script5 "https://github.com/damian666/literotica-dataset-creator/blob/master/data_P5_collate_stories.py"
    click enrich "https://github.com/damian666/literotica-dataset-creator/blob/master/enrich_metadata.py"
    click dup "https://github.com/damian666/literotica-dataset-creator/blob/master/remove_duplicates.py"
    click faulty "https://github.com/damian666/literotica-dataset-creator/blob/master/remove_faulty_entries.py"
    click val1 "https://github.com/damian666/literotica-dataset-creator/blob/master/validate.py"
    click val2 "https://github.com/damian666/literotica-dataset-creator/blob/master/validate_meta_structure.py"
    click sample "https://github.com/damian666/literotica-dataset-creator/blob/master/sample_dataset.py"

    %% Styles
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef extraction fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef post fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef storage fill:#E8E8E8,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
