How vardhan-r/yandere-scraper-and-cnn works
Project Type and Purpose
• This is a specialized data-collection and machine-learning tool rather than a full-stack web application.
• It has two main subsystems:
a. A high-throughput, concurrent web scraper for yande.re images
b. A TensorFlow-based Convolutional Neural Network (CNN) that classifies the scraped images
High-Level Architecture and Components
A. External Service
• yande.re HTTP API / HTML pages
B. Orchestration Layer (PowerShell)
• runner.ps1 – entry point for the scraping pipeline
• reset.ps1 – cleanup utility to delete images, CSVs, logs, next_post_id.txt
C. Scraper Core (Python)
• concurrent_scraper.py
– uses Python’s concurrent.futures (ThreadPoolExecutor)
– dispatches N worker threads to fetch posts concurrently
– calls BeautifulSoup parser (user-selectable parser like “lxml”)
– resizes images (e.g., via Pillow) to requested dimensions
• communication.txt / next_post_id.txt
– inter-run bookkeeping (last processed post, status messages)
D. Data Storage
• images/ directory (resized JPEG/PNG files)
• data.csv (metadata of successfully saved posts)
• did_not_save.csv (metadata of posts skipped due to missing data)
E. CNN Training and Evaluation
• cnn.ipynb
– loads images/ and data.csv
– constructs TensorFlow data pipeline (tf.data)
– builds and trains a CNN model, reports ~75% accuracy
Relationships and Data Flow
1. User invokes runner.ps1 with arguments (num_posts, max_workers, width, height, parser).
2. runner.ps1 launches concurrent_scraper.py with those parameters.
3. concurrent_scraper.py reads next_post_id.txt; starts ThreadPoolExecutor.
4. Each worker:
a. Makes HTTP request to yande.re/post/{id}
b. Parses HTML with BeautifulSoup → extracts image URL & metadata
c. Downloads raw image → resizes to WxH → saves under images/
d. Appends a row to data.csv or did_not_save.csv
e. Updates next_post_id.txt and logs to communication.txt
5. Once scraping is complete, images/ and CSVs contain the training corpus.
6. Data scientist opens cnn.ipynb → TF data pipeline reads images/ + labels from data.csv → trains CNN
Architectural Patterns and Principles
• Concurrency via thread pool: maximizes I/O throughput
• Separation of concerns: orchestration (PS1) vs. scraper logic (Python) vs. ML (notebook)
• Loose coupling: PowerShell scripts and Python code interact through CLI arguments and text files
• Reproducibility: reset.ps1 resets the entire environment to a clean slate