# vardhan-r/yandere-scraper-and-cnn architecture

> Architecture diagram of the GitHub repository vardhan-r/yandere-scraper-and-cnn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vardhan-r/yandere-scraper-and-cnn
- Repository: https://github.com/vardhan-r/yandere-scraper-and-cnn

## Overview

##### Project Type and Purpose
- This is a specialized data-collection and machine-learning tool rather than a full-stack web application.
- It has two main subsystems:
a. A high-throughput, concurrent web scraper for yande.re images
b. A TensorFlow-based Convolutional Neural Network (CNN) that classifies the scraped images

##### High-Level Architecture and Components
A. External Service
- yande.re HTTP API / HTML pages
B. Orchestration Layer (PowerShell)
- runner.ps1 – entry point for the scraping pipeline
- reset.ps1 – cleanup utility to delete images, CSVs, logs, next_post_id.txt
C. Scraper Core (Python)
- concurrent_scraper.py
– uses Python’s concurrent.futures (ThreadPoolExecutor)
– dispatches N worker threads to fetch posts concurrently
– calls BeautifulSoup parser (user-selectable parser like “lxml”)
– resizes images (e.g., via Pillow) to requested dimensions
- communication.txt / next_post_id.txt
– inter-run bookkeeping (last processed post, status messages)
D. Data Storage
- images/ directory (resized JPEG/PNG files)
- data.csv (metadata of successfully saved posts)
- did_not_save.csv (metadata of posts skipped due to missing data)
E. CNN Training and Evaluation
- cnn.ipynb
– loads images/ and data.csv
– constructs TensorFlow data pipeline (tf.data)
– builds and trains a CNN model, reports ~75% accuracy

##### Relationships and Data Flow
1. User invokes runner.ps1 with arguments (num_posts, max_workers, width, height, parser).
2. runner.ps1 launches concurrent_scraper.py with those parameters.
3. concurrent_scraper.py reads next_post_id.txt; starts ThreadPoolExecutor.
4. Each worker:
a. Makes HTTP request to yande.re/post/{id}
b. Parses HTML with BeautifulSoup → extracts image URL & metadata
c. Downloads raw image → resizes to WxH → saves under images/
d. Appends a row to data.csv or did_not_save.csv
e. Updates next_post_id.txt and logs to communication.txt
5. Once scraping is complete, images/ and CSVs contain the training corpus.
6. Data scientist opens cnn.ipynb → TF data pipeline reads images/ + labels from data.csv → trains CNN

##### Architectural Patterns and Principles
- Concurrency via thread pool: maximizes I/O throughput
- Separation of concerns: orchestration (PS1) vs. scraper logic (Python) vs. ML (notebook)
- Loose coupling: PowerShell scripts and Python code interact through CLI arguments and text files
- Reproducibility: reset.ps1 resets the entire environment to a clean slate

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Service
    YANDE["yande.re Service"]:::external

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        RUNNER["runner.ps1"]:::scripts
        RESET["reset.ps1"]:::scripts
    end

    %% Core Scraper
    subgraph "Core Scraper"
        SCRAPER["concurrent_scraper.py"]:::modules
        subgraph "Libraries"
            PARSER["BeautifulSoup"]:::modules
            RESIZER["Pillow"]:::modules
        end
        BOOKKEEP["next_post_id.txt"]:::datastore
        LOG["communication.txt"]:::datastore
    end

    %% Data Storage
    subgraph "Data Storage"
        IMAGES["images/"]:::datastore
        SAVED_CSV["data.csv"]:::datastore
        SKIPPED_CSV["did_not_save.csv"]:::datastore
        subgraph "Historical Data"
            HIST_SAVED["scraped data/data_upto_131000.csv"]:::datastore
            HIST_SKIPPED["scraped data/did_not_save_upto_131000.csv"]:::datastore
        end
    end

    %% ML Pipeline
    subgraph "ML Pipeline"
        NOTEBOOK["cnn.ipynb"]:::ml
    end

    %% Connections
    RUNNER -->|"launches"| SCRAPER
    RUNNER -->|"cleans env"| RESET
    SCRAPER -->|reads/writes| BOOKKEEP
    SCRAPER -->|logs status| LOG
    SCRAPER -->|HTTP GET posts| YANDE
    SCRAPER -->|parses HTML| PARSER
    PARSER -->|metadata & URL| SCRAPER
    SCRAPER -->|downloads image| RESIZER
    RESIZER -->|resized image| IMAGES
    SCRAPER -->|append row| SAVED_CSV
    SCRAPER -->|append row| SKIPPED_CSV
    IMAGES -->|read images| NOTEBOOK
    SAVED_CSV -->|read labels| NOTEBOOK

    %% Click Events
    click RUNNER "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/runner.ps1"
    click RESET "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/reset.ps1"
    click SCRAPER "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/concurrent_scraper.py"
    click BOOKKEEP "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/next_post_id.txt"
    click LOG "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/communication.txt"
    click IMAGES "https://github.com/vardhan-r/yandere-scraper-and-cnn/tree/main/images/"
    click SAVED_CSV "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/data.csv"
    click SKIPPED_CSV "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/did_not_save.csv"
    click HIST_SAVED "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/scraped data/data_upto_131000.csv"
    click HIST_SKIPPED "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/scraped data/did_not_save_upto_131000.csv"
    click NOTEBOOK "https://github.com/vardhan-r/yandere-scraper-and-cnn/blob/main/cnn.ipynb"

    %% Styles
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef scripts fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef modules fill:#90EE90,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFD700,stroke:#333,stroke-width:1px
    classDef ml fill:#DDA0DD,stroke:#333,stroke-width:1px
```
