# stanfordsnr/puffer architecture

> Architecture diagram of the GitHub repository stanfordsnr/puffer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stanfordsnr/puffer
- Repository: https://github.com/stanfordsnr/puffer

## Overview

##### Project Type and Purpose
- Puffer is a full-stack, live-TV streaming platform (Stanford research + public website).
- It consists of:
– A high-performance C++ streaming pipeline (input ingestion, decoding, packaging, serving)
– A Python/Django “portal” web application for user accounts, front-end UI, and playback pages
– A set of auxiliary scripts (Python) for data processing, monitoring, and maintenance
– A monitoring subsystem (InfluxDB metrics, reporters)

##### File Structure
- Top-level “src” directory breaks into many C++ modules, each with its own Makefile.am:
– abr: adaptive–bitrate algorithms (BOLA, Pensieve, MPC, etc.)
– atsc: MPEG-TS decoder for live broadcast
– forwarder: UDP→TCP, file sender/receiver
– cleaner: content-sanitizing modules
– media-server: channel multiplexing, WebSocket server
– mp4, webm, mpd: packaging libraries and MPD writer
– net: HTTP/WebSocket stack, sockets, parsers
– notifier: inotify wrapper for file events
– wrappers: command wrappers to drive encoding/fragmentation pipelines
- The “portal” folder is a Django project with two apps:
– accounts: user registration/login
– puffer: rating, player UI, participation pages
- “scripts” are Python tools for snapshot generation, influxdb backup, stream processing, etc.
- “monitoring” collects and reports metrics via InfluxDB clients and config files.

##### Extract Architecture Clues from README
- Real-time live-TV ingestion → processed by ML-driven ABR → delivered via HTTP/WebSocket to browser players
- Web portal is the customer-facing UI where users log in, watch streams, rate them, and view monitoring dashboards

##### Main Components & Their Interactions
a. Source Ingestion
– Live TV sources (ATSC, file replays, synthetic streams) feed into the “forwarder” module.
b. Pre-Processing & Cleaning
– forwarder routes streams into “cleaner”, then into the “atsc” decoder.
c. Packaging & ABR
– Decoded streams pass through “wrappers” and packaging modules (mp4, webm, mpd).
– At packaging time or in the client, ABR modules (C++ abr/*) decide which bitrate chunk to serve.
– Python IPC bridge connects C++ ABR logic to any Python‐based orchestrators.
d. Media Server
– media-server hosts channels, serves segments over HTTP or WebSocket (ws_media_server).
– Clients connect via secure sockets (net/secure_socket, ws_server).
e. Web Portal (Django)
– Handles user accounts, serves HTML/JS front-end for video player and ratings.
– Stores metadata and ratings in a relational database (e.g., SQLite/Postgres via Django ORM).
– Static files (player.js, CSS) delivered by Django’s static-files mechanism.
f. Monitoring & Reporting
– Components instrumented to emit metrics (active_streams, client_buffer, video_sent, etc.).
– monitoring/file_reporter and influxdb_client push data into an InfluxDB time-series database.
– Grafana (external) visualizes these metrics via webhooks and snapshot scripts.
g. Management & Maintenance Scripts
– Python scripts under src/scripts handle backups, restore_influxdb, stream_processor, mock_decoder, etc.

##### Architectural Patterns & Design Principles
- Modular, Makefile-based architecture: each subsystem compiles independently.
- Separation of concerns:
– C++ pipeline for performance-critical streaming logic
– Python/Django for business logic, web UI, and ease of iteration
– Scripts for ops, data processing, monitoring orchestration
- IPC & wrapping: command-line wrappers and Python IPC bridge allow cross-language integration.
- Event-driven monitoring: inotify + file-based reporters → time-series db

##### Final Touches
- Arrowheads denote direction of data flow or request/response.
- Keep labels concise but informative (module name + role).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion & Processing
    subgraph "Ingestion & Processing"
        direction TB
        LiveTVSource["Live TV Sources<br/>(ATSC, File, Synthetic)"]:::external
        Forwarder["forwarder<br/>(C++, UDP→TCP)"]:::ingestion
        Cleaner["cleaner<br/>(C++, Sanitizer)"]:::ingestion
        ATSCDecoder["atsc<br/>(C++, MPEG-TS Decoder)"]:::ingestion
    end

    LiveTVSource -->|UDP| Forwarder
    Forwarder -->|stream| Cleaner
    Cleaner -->|cleaned stream| ATSCDecoder

    %% Packaging & ABR
    subgraph "Packaging & ABR"
        direction TB
        Wrappers["wrappers<br/>(C++, CLI Wrappers)"]:::packaging
        ABR["abr<br/>(C++, ABR Algorithms)"]:::packaging
        PythonIPC["Python IPC Bridge"]:::packaging
        MP4["mp4<br/>(C++, MP4 Packaging)"]:::packaging
        WebM["webm<br/>(C++, WebM Packaging)"]:::packaging
        MPD["mpd<br/>(C++, MPD Writer)"]:::packaging
        Util["util<br/>(C++, Utility Lib)"]:::packaging
        Opus["opus-encoder<br/>(C++, Opus Encoder)"]:::packaging
    end

    ATSCDecoder -->|decoded| Wrappers
    Wrappers --> Util
    Wrappers --> Opus
    Wrappers --> MP4
    Wrappers --> WebM
    Wrappers --> MPD
    ABR -->|bitrate decisions| Wrappers
    ABR -->|decisions| PythonIPC
    PythonIPC --> ABR

    %% Media Server
    subgraph "Media Server"
        direction TB
        Networking["net<br/>(C++, HTTP/WebSocket)"]:::media
        MediaServer["media-server<br/>(C++, WS Server)"]:::media
    end

    MP4 -->|segments| MediaServer
    WebM -->|segments| MediaServer
    MPD -->|manifests| MediaServer
    MediaServer --> Networking
    Networking -->|HTTP/WS| Client

    %% Client Browser
    Client["Browser Player"]:::external

    %% Web Portal
    subgraph "Web Portal (Django)"
        direction TB
        PortalAccounts["accounts app"]:::portal
        PortalPuffer["puffer app"]:::portal
        PortalSettings["portal/settings.py"]:::portal
        PortalDB["Django DB<br/>(ORM)"]:::database
    end

    Client -->|HTTP| PortalAccounts
    Client -->|HTTP| PortalPuffer
    PortalAccounts -->|SQL| PortalDB
    PortalPuffer -->|SQL| PortalDB

    %% Monitoring & Reporting
    subgraph "Monitoring & Reporting"
        direction TB
        Notifier["notifier<br/>(C++, inotify)"]:::monitor
        FileReporter["file_reporter<br/>(Python)"]:::monitor
        InfluxClient["influxdb_client<br/>(Python)"]:::monitor
        InfluxDB["InfluxDB"]:::db
        Grafana["Grafana"]:::external
    end

    MediaServer -->|metrics| FileReporter
    Notifier -->|file events| FileReporter
    FileReporter --> InfluxClient
    InfluxClient --> InfluxDB
    Grafana -->|queries| InfluxDB

    %% Scripts & Batch Jobs
    subgraph "Scripts & Batch Jobs"
        direction TB
        Scripts["scripts<br/>(Python Tools)"]:::scripts
    end

    Scripts -->|backup/restore| InfluxDB
    Scripts -->|process streams| MediaServer
    Scripts -->|filesystem ops| Notifier

    %% Click Events
    click Forwarder "https://github.com/stanfordsnr/puffer/tree/master/src/forwarder/"
    click Cleaner "https://github.com/stanfordsnr/puffer/tree/master/src/cleaner/"
    click ATSCDecoder "https://github.com/stanfordsnr/puffer/tree/master/src/atsc/"
    click MP4 "https://github.com/stanfordsnr/puffer/tree/master/src/mp4/"
    click WebM "https://github.com/stanfordsnr/puffer/tree/master/src/webm/"
    click MPD "https://github.com/stanfordsnr/puffer/tree/master/src/mpd/"
    click ABR "https://github.com/stanfordsnr/puffer/tree/master/src/abr/"
    click Wrappers "https://github.com/stanfordsnr/puffer/tree/master/src/wrappers/"
    click PythonIPC "https://github.com/stanfordsnr/puffer/blob/master/src/abr/python_ipc.py"
    click Util "https://github.com/stanfordsnr/puffer/tree/master/src/util/"
    click Opus "https://github.com/stanfordsnr/puffer/tree/master/src/opus-encoder/"
    click Networking "https://github.com/stanfordsnr/puffer/tree/master/src/net/"
    click MediaServer "https://github.com/stanfordsnr/puffer/tree/master/src/media-server/"
    click Notifier "https://github.com/stanfordsnr/puffer/tree/master/src/notifier/"
    click InfluxClient "https://github.com/stanfordsnr/puffer/tree/master/src/monitoring/"
    click Scripts "https://github.com/stanfordsnr/puffer/tree/master/src/scripts/"
    click PortalSettings "https://github.com/stanfordsnr/puffer/blob/master/src/portal/portal/settings.py"
    click PortalAccounts "https://github.com/stanfordsnr/puffer/tree/master/src/portal/accounts/"
    click PortalPuffer "https://github.com/stanfordsnr/puffer/tree/master/src/portal/puffer/"

    %% Styles
    classDef ingestion fill:#FFEEEE,stroke:#CC0000
    classDef packaging fill:#EEFFEE,stroke:#00AA00
    classDef media fill:#EEEEFF,stroke:#0000CC
    classDef portal fill:#FFF0E0,stroke:#CC6600
    classDef database fill:#FFFFCC,stroke:#CCCC00
    classDef monitor fill:#F0F0F0,stroke:#888888
    classDef scripts fill:#E8E8FF,stroke:#6666CC
    classDef external fill:#FFFFFF,stroke:#333333,stroke-dasharray: 5 5
```
