How stanfordsnr/puffer works
Project Type and Purpose
• Puffer is a full-stack, live-TV streaming platform (Stanford research + public website).
• It consists of:
– A high-performance C++ streaming pipeline (input ingestion, decoding, packaging, serving)
– A Python/Django “portal” web application for user accounts, front-end UI, and playback pages
– A set of auxiliary scripts (Python) for data processing, monitoring, and maintenance
– A monitoring subsystem (InfluxDB metrics, reporters)
File Structure
• Top-level “src” directory breaks into many C++ modules, each with its own Makefile.am:
– abr: adaptive–bitrate algorithms (BOLA, Pensieve, MPC, etc.)
– atsc: MPEG-TS decoder for live broadcast
– forwarder: UDP→TCP, file sender/receiver
– cleaner: content-sanitizing modules
– media-server: channel multiplexing, WebSocket server
– mp4, webm, mpd: packaging libraries and MPD writer
– net: HTTP/WebSocket stack, sockets, parsers
– notifier: inotify wrapper for file events
– wrappers: command wrappers to drive encoding/fragmentation pipelines
• The “portal” folder is a Django project with two apps:
– accounts: user registration/login
– puffer: rating, player UI, participation pages
• “scripts” are Python tools for snapshot generation, influxdb backup, stream processing, etc.
• “monitoring” collects and reports metrics via InfluxDB clients and config files.
Extract Architecture Clues from README
• Real-time live-TV ingestion → processed by ML-driven ABR → delivered via HTTP/WebSocket to browser players
• Web portal is the customer-facing UI where users log in, watch streams, rate them, and view monitoring dashboards
Main Components & Their Interactions
a. Source Ingestion
– Live TV sources (ATSC, file replays, synthetic streams) feed into the “forwarder” module.
b. Pre-Processing & Cleaning
– forwarder routes streams into “cleaner”, then into the “atsc” decoder.
c. Packaging & ABR
– Decoded streams pass through “wrappers” and packaging modules (mp4, webm, mpd).
– At packaging time or in the client, ABR modules (C++ abr/*) decide which bitrate chunk to serve.
– Python IPC bridge connects C++ ABR logic to any Python‐based orchestrators.
d. Media Server
– media-server hosts channels, serves segments over HTTP or WebSocket (ws_media_server).
– Clients connect via secure sockets (net/secure_socket, ws_server).
e. Web Portal (Django)
– Handles user accounts, serves HTML/JS front-end for video player and ratings.
– Stores metadata and ratings in a relational database (e.g., SQLite/Postgres via Django ORM).
– Static files (player.js, CSS) delivered by Django’s static-files mechanism.
f. Monitoring & Reporting
– Components instrumented to emit metrics (active_streams, client_buffer, video_sent, etc.).
– monitoring/file_reporter and influxdb_client push data into an InfluxDB time-series database.
– Grafana (external) visualizes these metrics via webhooks and snapshot scripts.
g. Management & Maintenance Scripts
– Python scripts under src/scripts handle backups, restore_influxdb, stream_processor, mock_decoder, etc.
Architectural Patterns & Design Principles
• Modular, Makefile-based architecture: each subsystem compiles independently.
• Separation of concerns:
– C++ pipeline for performance-critical streaming logic
– Python/Django for business logic, web UI, and ease of iteration
– Scripts for ops, data processing, monitoring orchestration
• IPC & wrapping: command-line wrappers and Python IPC bridge allow cross-language integration.
• Event-driven monitoring: inotify + file-based reporters → time-series db
Final Touches
• Arrowheads denote direction of data flow or request/response.
• Keep labels concise but informative (module name + role).