# criticalsession/4scraper architecture

> Architecture diagram of the GitHub repository criticalsession/4scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/criticalsession/4scraper
- Repository: https://github.com/criticalsession/4scraper

## Overview

##### External Dependencies (Show as external systems)
- 4chan API/Website
- Local File System
- Colly (Web Scraping Library)
- ProgressBar Library

##### Data Flow and Process Steps

**[User Input] → [CLI Interface]**
- ↓
**[Config Manager] ← [Core Controller] → [Search Service]**
- ↓
**[Scraper Engine] → [4chan Website]**
- ↓
**[Downloader Service] → [File Manager]**
- ↓
**[Local File System]**

Key Architectural Elements to Highlight:

##### Input Processing Layer
- How flags and URL inputs are processed
- To config parsing functionality

##### Core Processing Layer
- Scraping pipeline
- Parallel download management
- Demonstrate search functionality

##### Storage Layer
- File system interaction
- Demonstrate directory structure management

Design Guidelines:

##### Specific Details
- Configuration flow
- Parallel download management
- File naming and directory structure logic
- Search functionality pipeline

##### Configurable aspects
- Board directory structure
- Thread directory structure
- Original filename preservation
- File extension filtering

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "User Interface Layer"
        CLI["CLI Interface"]
        FLAGS["Flags/Args Handler"]:::config
        INTERACTIVE["Interactive Input"]
    end

    subgraph "Core Processing Layer"
        CONFIG["Config Manager"]:::config
        SCRAPER["Scraper Engine"]:::scraper
        DOWNLOADER["Downloader Service"]:::downloader
        SEARCH["Search Service"]:::search
        FILEMGR["File Manager"]:::filemgr
    end

    subgraph "External Dependencies"
        CHAN["4chan API/Website"]
        FILESYSTEM["Local File System"]
        COLLY["Colly Library"]
        PROGRESS["ProgressBar Library"]
    end

    %% Connections
    CLI --> FLAGS
    CLI --> INTERACTIVE
    FLAGS --> CONFIG
    INTERACTIVE --> CONFIG
    CONFIG --> SCRAPER
    CONFIG --> DOWNLOADER
    CONFIG --> SEARCH
    SCRAPER --> CHAN
    SCRAPER --> COLLY
    DOWNLOADER --> PROGRESS
    DOWNLOADER --> FILEMGR
    SEARCH --> FILEMGR
    FILEMGR --> FILESYSTEM

    %% Parallel Processing Annotation
    subgraph "Concurrent Downloads"
        PARALLEL["Max 20 Threads"]
    end
    DOWNLOADER --- PARALLEL

    %% Legend
    subgraph "Legend"
        L1["UI Component"]:::ui
        L2["Core Component"]:::core
        L3["External System"]:::external
        L4["→ Direct Call"]
        L5["--- Annotation"]
    end

    %% Styles
    classDef ui fill:#a8d1f0
    classDef core fill:#90EE90
    classDef external fill:#FFE5B4
    classDef config fill:#D8BFD8
    classDef scraper fill:#98FB98
    classDef downloader fill:#87CEEB
    classDef search fill:#DDA0DD
    classDef filemgr fill:#F0E68C

    %% Click Events
    click CONFIG "https://github.com/criticalsession/4scraper/tree/main/config/"
    click FLAGS "https://github.com/criticalsession/4scraper/blob/main/config/flags.go"
    click SCRAPER "https://github.com/criticalsession/4scraper/blob/main/scraper.go"
    click DOWNLOADER "https://github.com/criticalsession/4scraper/blob/main/downloader.go"
    click SEARCH "https://github.com/criticalsession/4scraper/tree/main/search/"
    click FILEMGR "https://github.com/criticalsession/4scraper/tree/main/file/"
```
