# internetarchive/brozzler architecture

> Architecture diagram of the GitHub repository internetarchive/brozzler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internetarchive/brozzler
- Repository: https://github.com/internetarchive/brozzler

## Overview

##### Main Components

a) Core Components:
- Chrome/Chromium Browser Instance
- RethinkDB (Database)
- Warcprox (Web Archive Proxy)
- Brozzler Worker(s)
- Brozzler Dashboard
- PyWB (Wayback Machine)

b) Supporting Components:
- YT-DLP Integration
- X11 Environment (Xvnc4/Xvfb)
- Service Registry

##### Component Relationships and Data Flow

Frontend Layer:
- Brozzler Dashboard (Port 8000)
- PyWB Wayback Interface (Port 8880)

Processing Layer:
- Brozzler Workers (Multiple instances possible)
- Chrome/Chromium Browser instances
- YT-DLP Media Processor

Storage Layer:
- RethinkDB (Crawl state management)
- WARC Files Storage

##### Specific Implementation Details

- Rectangles: Core services (Dashboard, Workers, Browser)
- Hexagons: External dependencies (Chrome, YT-DLP)

##### Component Details

Dashboard:

- Brozzler Dashboard |
- Job Management   |
- Status Viewing   |
- Port: 8000        |

Worker Architecture:

- Brozzler Worker     |
- Browser Control      |
- URL Queue Management |
- Media Capture        |

Database Schema Hints:
**RethinkDB**
- Crawl State
- Service Registry
- Captures Table

##### Deployment View

How components can be:
- Run together (brozzler-easy mode)
- Distributed across machines
- Scaled horizontally

- Distributed nature of the system
- Browser-based crawling approach
- Integration with web archiving tools
- Real-time playback capabilities

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Frontend
        Dashboard["Brozzler Dashboard<br/>(Port 8000)"]
        PyWB["PyWB Wayback<br/>(Port 8880)"]
        UI["Frontend UI"]
    end

    subgraph Processing
        Worker["Brozzler Worker"]
        Browser["Chrome/Chromium<br/>Browser Instance"]
        YTDL["YT-DLP Media<br/>Processor"]
        X11["X11 Environment<br/>(Xvnc4/Xvfb)"]
    end

    subgraph Storage
        RethinkDB[(RethinkDB)]
        WARC[("WARC Files<br/>Storage")]
        JobQueue["Job/URL Queue"]
    end

    subgraph Components
        Warcprox["Warcprox Proxy"]
        ServiceReg["Service Registry"]
    end

    %% Relationships
    Dashboard --> RethinkDB
    Dashboard --> JobQueue
    UI --> Dashboard
    Worker --> Browser
    Worker --> YTDL
    Worker --> JobQueue
    Browser --> Warcprox
    Warcprox --> WARC
    PyWB --> WARC
    X11 --> Browser
    ServiceReg --> RethinkDB

    %% Click Events
    click Dashboard "https://github.com/internetarchive/brozzler/tree/master/brozzler/dashboard/"
    click Worker "https://github.com/internetarchive/brozzler/blob/master/brozzler/worker.py"
    click Browser "https://github.com/internetarchive/brozzler/blob/master/brozzler/chrome.py"
    click YTDL "https://github.com/internetarchive/brozzler/blob/master/brozzler/ydl.py"
    click X11 "https://github.com/internetarchive/brozzler/blob/master/ansible/roles/brozzler-worker/templates/Xvnc-run.j2"
    click RethinkDB "https://github.com/internetarchive/brozzler/blob/master/brozzler/model.py"
    click Warcprox "https://github.com/internetarchive/brozzler/tree/master/ansible/roles/warcprox/"
    click PyWB "https://github.com/internetarchive/brozzler/blob/master/brozzler/pywb.py"
    click UI "https://github.com/internetarchive/brozzler/tree/master/brozzler/dashboard/static/"
    click JobQueue "https://github.com/internetarchive/brozzler/blob/master/brozzler/frontier.py"

    %% Styling
    classDef frontend fill:#a8d5ff
    classDef processing fill:#ffb3b3
    classDef storage fill:#b3ffb3
    classDef component fill:#ffe6b3

    class Dashboard,PyWB,UI frontend
    class Worker,Browser,YTDL,X11 processing
    class RethinkDB,WARC,JobQueue storage
    class Warcprox,ServiceReg component

    %% Legend
    subgraph Legend
        Frontend_L["Frontend Components"]:::frontend
        Processing_L["Processing Components"]:::processing
        Storage_L["Storage Components"]:::storage
        Component_L["Supporting Components"]:::component
    end
```
