How ayamir/tiled-vr-dash-platform works
Project Type and Purpose
• The project is a full-scale video streaming platform that incorporates both server-side and client-side components.
• It handles video processing – primarily cutting, transcoding (using H.264), and DASH packaging on the server – while on the client side, it supports two different players: one for PC (react-xrplayer) and one for VR headsets (webxr-player). The VR player even supports viewport prediction (VP), whereas the PC variant does not.
• Additionally, the platform contains modules for viewport prediction (vp), emulation of network conditions (emulate), and analysis of client-generated JSON data (analysis).
File Structure and Map out Components
• Server Components:
• stream: Contains scripts (e.g., crop_video.py, dash_tiles.py, multi_version.py) for video segmentation, encoding, and possibly live DASH streaming. It even includes a README and assets that suggest its role is to prepare video content for delivery.
• vp: Provides viewport prediction functionalities, indicated by directories like “trace-based-fov-predict.”
• emulate: Simulates network conditions (with subdirectories for “4g” and “5g” emulation), likely leveraging real-world tc command data.
• analysis: Processes JSON data coming from the client to generate graphs or visual insights using Python scripts (cmder.py, deploy.py, visual.py).
• Client Components:
• react-xrplayer: A single-page application (SPA) for PC-based video viewing that does not include VP prediction.
• webxr-player: A more complex client tailored for VR headsets (like Meta Quest 2), consisting of several detailed subdirectories such as:
▪ css for styling,
▪ js libraries for various functionalities including DASH integration, matrix math, 3D rendering, and scene management.
▪ Third-party libraries like gl-matrix and three.js for WebGL rendering, as well as the webxr-polyfill to ensure proper XR support.
▪ Utility scripts and experimental modules (such as hit-test.js, stereo-util.js) that support VR interactions.
Main Components and Their Interactions
• Server Layer:
• Video Processor (stream module): Cuts and transcodes original videos; outputs DASH-compatible segments.
• Viewport Predictor (vp module): Uses trace-based algorithms to estimate the Viewport (VP) for VR optimizations.
• Network Emulator (emulate module): Simulates different network bandwidths using real network data to test the system’s resilience.
• Data Analyzer (analysis module): Receives client-collected JSON data, analyzes it, and produces visualizations or performance metrics.
• Client Layer:
• PC Viewer (react-xrplayer): Connects to the streaming server and receives video streams for PC.
• VR Player (webxr-player):
▪ Rendering Engine: Uses matrix libraries, shader programs, and WebGL renderers (found in subdirectories like render/core, render/geometry, etc.) to display video in immersive environments.
▪ Interactivity & Utility: Includes modules for handling inputs, hit tests, and UI elements (found in js/render/nodes).
▪ Third-Party Dependencies: Integrates with libraries like three.js and gl-matrix to handle 3D operations.
• Data Flow and Interaction:
▪ The video processing server sends the DASH-packaged video to both client types.
▪ The VR player can leverage the viewport prediction results from the vp module to optimize video delivery.
▪ The analysis module gets feedback data (JSON format) from client sessions, which helps in system performance analysis.
Architectural Patterns and Key Technologies
• Separation of Concerns: The clear partition between server modules (stream, vp, emulate, analysis) and client modules (react-xrplayer, webxr-player) highlights a layered architecture.
• Modularity: Each server module focuses on a specific function (video processing, prediction, emulation, analysis) while the client’s structure separates basic video playback from advanced VR features.
• Technology Stack:
▪ Video Encoding: H.264.
▪ Video Streaming: DASH.
▪ Frontend Libraries: React (react-xrplayer) and WebXR with three.js, gl-matrix, and webxr-polyfill (webxr-player).
▪ Server-Side Tools: Python scripts for analysis, emulation, and streaming.
▪ Additional: Use of matrix math libraries and rendering pipelines in the VR player.