# archiveteam/grab-site architecture

> Architecture diagram of the GitHub repository archiveteam/grab-site, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/archiveteam/grab-site
- Repository: https://github.com/archiveteam/grab-site

## Overview

##### Main Components

a) Core Components:
- Dashboard Server (gs-server)
- Crawler Engine (based on wpull)
- URL Queue Manager
- Ignore Pattern Manager
- WARC File Generator
- Duplicate Detection System

b) Storage Components:
- SQLite Database (for URL queue)
- WARC Files Storage
- Control Files Storage

##### Component Relationships and Data Flow

- User → Dashboard (HTTP/WebSocket)
- Dashboard ↔ Crawler Instances (WebSocket)
- Crawler → URL Queue Manager (bidirectional)
- Crawler → Target Websites (HTTP requests)
- Crawler → WARC Generator (content streaming)
- Crawler → Ignore Pattern Manager (URL filtering)
- Crawler → Duplicate Detection (content analysis)

##### Architectural Pattern
Use a distributed architecture pattern showing:
- Multiple crawler instances can run simultaneously
- Central dashboard for monitoring
- Independent storage systems

##### Component Details

Dashboard Server:

- gs-server         |

- WebSocket Server     |
- HTTP Server          |
- Crawl Monitor        |
- Status Updates       |

Crawler Engine:

- Crawler Instance    |

- wpull Engine         |
- HTTP Client          |
- Link Extractor       |
- Content Processor    |

##### Key Technical Elements to Highlight
- Python 3.7/3.8 runtime
- WebSocket communication
- SQLite database for queue
- WARC file format
- Regular expressions for URL filtering
- Disk-based queue system

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph User["User Interface"]
        UI[Web Browser]
        CLI[Command Line]
    end

    subgraph Dashboard["Dashboard Layer"]
        DS["Dashboard Server"]:::server
        DH["Dashboard HTML"]:::frontend
        DC["Dashboard Client"]:::client
    end

    subgraph Core["Core Processing Layer"]
        CE["Crawler Engine"]:::engine
        WH["Wpull Hooks"]:::engine
        WT["Wpull Tweaks"]:::engine
        MAIN["Main Controller"]:::controller
    end

    subgraph Storage["Storage Layer"]
        DB[(SQLite Database)]
        WARC[("WARC Files")]
    end

    subgraph Control["Control Systems"]
        IPM["Ignore Pattern Manager"]:::manager
        DDS["Duplicate Detection"]:::detector
        URLS["URL Manager"]:::manager
    end

    subgraph Config["Configuration"]
        COOK["Default Cookies"]:::config
        INIT["Init Config"]:::config
    end

    subgraph Error["Error Handling"]
        ERR["404 Handler"]:::error
    end

    %% Relationships
    UI --> DS
    CLI --> MAIN
    DS --> CE
    CE --> WH
    CE --> WT
    WH --> DB
    WH --> WARC
    CE --> IPM
    CE --> DDS
    CE --> URLS
    DS --> DH
    DS --> DC

    %% Click Events
    click DS "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/server.py"
    click DH "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/dashboard.html"
    click DC "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/dashboard_client.py"
    click WH "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/wpull_hooks.py"
    click WT "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/wpull_tweaks.py"
    click MAIN "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/main.py"
    click IPM "https://github.com/ArchiveTeam/grab-site/tree/master/libgrabsite/ignore_sets/"
    click DDS "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/dupes.py"
    click URLS "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/dump_urls.py"
    click COOK "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/default_cookies.txt"
    click INIT "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/__init__.py"
    click ERR "https://github.com/ArchiveTeam/grab-site/blob/master/libgrabsite/404.html"

    %% Styles
    classDef server fill:#3498db
    classDef frontend fill:#2ecc71
    classDef client fill:#e74c3c
    classDef engine fill:#9b59b6
    classDef controller fill:#34495e
    classDef manager fill:#f1c40f
    classDef detector fill:#e67e22
    classDef config fill:#1abc9c
    classDef error fill:#c0392b

    %% Legend
    subgraph Legend
        L1["Server Component"]:::server
        L2["Frontend Component"]:::frontend
        L3["Client Component"]:::client
        L4["Engine Component"]:::engine
        L5["Control Component"]:::controller
        L6["Manager Component"]:::manager
        L7["Detection Component"]:::detector
        L8["Config Component"]:::config
        L9["Error Component"]:::error
    end
```
