# axiscommunications/media-stream-library-js architecture

> Architecture diagram of the GitHub repository axiscommunications/media-stream-library-js, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/axiscommunications/media-stream-library-js
- Repository: https://github.com/axiscommunications/media-stream-library-js

## Overview

1. The "streams" layer – a set of low-level components and pipelines that convert RTP (H.264/AAC or JPEG) or HTTP-based MP4 streams into ISO-BMFF and feed them into a browser’s Media Source Extensions (MSE). This includes various submodules for handling RTSP (parsing, authentication, depayloading, and session management), as well as MP4 muxing/parsing.

2. The "player" layer – which is provided as a React-based player component (with variations such as BasicPlayer and an advanced Player) intended to present the video state using dedicated React components (controls, feedback, image overlays, etc.). There are examples for both React (example-player-react) and Web Components (example-player-webcomponent).

3. The "overlay" module – a React-based library mainly intended for coordinate conversions for drawing SVG overlays. It provides a “Foundation” component for transforming between user and SVG coordinate systems, along with hooks and components like “Liner” and “useDraggable”.

4. Supplementary example applications and test harnesses – many example directories (for overlay, player, streams) demonstrate how end-user applications can integrate with the library. There are also scripts (e.g. using Docker, justfiles, or Vite) to facilitate local testing and CI integration.

5. Supporting infrastructure – GitHub workflows (CI, Docker, dependency updates), ESLint/TypeScript configurations, and other developer tools ensuring code quality and constant integration.

A. Main Components:
- Client (Browser): Represents the end-user environment running the player and overlay components. This includes the React components and web components consuming the streams.
- Media Stream Pipelines: Represent the low-level streaming “engines” contained under the /src/streams directory. This should include separate submodules for:
- RTP stream handling (e.g. RTSP parsing, RTP depayloading)
- MP4 capture and HTTP-MP4 pipeline for low latency
- Utility components for muxing/demuxing and conversion purposes.
- Player Module: Contained under /src/player and the corresponding example applications (example-player-react, example-player-webcomponent). This module handles the presentation of video, extraction of video state, and user interactions.
- Overlay Module: The /src/overlay components that abstract coordinate conversions, dealing with user/SVG coordinate systems. Show its relationship with visual output on top of the video.
- External Camera/RTSP Source: This is the external dependency represented by an Axis IP camera. It provides the original stream and sometimes expects two-way communications over WebSocket.
- Development/Build Tools and CI/CD: Represented by the configuration files (.github/workflows, justfile, esbuild.mjs, etc.), which manage automated testing, building, and deployment.

B. Relationships and Interactions:
- Within the pipelines, indicate sub-components (like RTSP authentication and parsing, RTP depayloading, MP4 muxing) that feed into the “Player Module.”
- The “Player Module” then provides media data to the Client (Browser) environment, which uses MSE to render the video.
- The “Overlay Module” sits as an additional layer on top of the video feed, taking cues from user-defined coordinate systems and transformation matrices to place SVG or other overlay components.
- That examples and test harnesses serve as demonstrative “front-end” clients that directly integrate with these technologies.

C. Architectural Patterns and Principles:
- Modular design: All concerns (stream processing, playback, overlay) are separated into distinct modules (directories) to allow independent development and potential reuse.
- Pipeline pattern: The streaming components align as a pipeline (or chain of responsibility) where each component processes or transforms media data.
- React-based view composition: The player and overlay layers follow a component-based architecture characteristic of React applications.
- External dependency integration: The design accounts for two-way communication with the RTSP server via WebSocket, which is crucial for playback synchronization.

D. Relevant Technologies / Frameworks:
- JavaScript/TypeScript (core language for all modules, as indicated by .ts and .tsx files).
- React (for the player and overlay).
- Web Streams API and Media Source Extensions (for dynamic stream feeding into browsers).
- Node.js and Yarn (for development and package management).
- Vite (for development server in the example React applications).
- Docker (for the gst-rtsp-launch integration and deployment testing).

- External integrations (Axis camera, WebSocket communications, HTTP-MP4 pipelines) with concise labels.
- Optionally, include a “Developer/CI/CD” block that shows the integration of GitHub Workflows, justfile, and various configuration files.

F. Tailoring to This Project Type:
- Since this is a media streaming library with both full-stack (client-server interaction) and specialized front-end rendering (React-based player and overlay) capabilities, emphasize the distinction between data acquisition (stream ingestion & pipeline processing) and stateful UI rendering (player and overlay).
- To detail how the modularity (with clearly separated subdirectories and examples) supports extensibility in building custom player or overlay solutions.

This level of granularity will capture the essential architectural elements and serve as an accurate system design blueprint for the Media Stream Library JS project.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Source
    EC("Axis IP Camera"):::external

    %% Media Stream Pipelines Subgraph
    subgraph "Media Stream Pipelines"
        RTP("RTP Handling"):::backend
        RTSP("RTSP Handling"):::backend
        RTSP_AUTH("RTSP Auth"):::backend
        MP4MUX("MP4 Muxer"):::backend
        MP4PAR("MP4 Parser"):::backend
        HTTP_MP4("HTTP MP4 Pipeline"):::backend
        RTSP_MP4("RTSP MP4 Pipeline"):::backend
        MSP("Media Stream Output"):::backend
    end

    %% Connect external camera to streaming pipelines nodes
    EC -->|"RTSP/HTTP Stream"| RTP
    EC -->|"RTSP/HTTP Stream"| RTSP

    %% Connect RTSP subcomponent
    RTSP --> RTSP_AUTH

    %% All pipeline components feed into stream output
    RTP --> MSP
    RTSP --> MSP
    RTSP_AUTH --> MSP
    MP4MUX --> MSP
    MP4PAR --> MSP
    HTTP_MP4 --> MSP
    RTSP_MP4 --> MSP

    %% Player Module Subgraph
    subgraph "Player Module"
        BP("BasicPlayer Component"):::frontend
        PC("Player Component"):::frontend
        EX_REACT("Example Player: React"):::frontend
        EX_WEB("Example Player: WebComponent"):::frontend
    end

    %% Overlay Module Subgraph
    subgraph "Overlay Module"
        OF("Overlay: Foundation"):::frontend
        OL("Overlay: Liner"):::frontend
        OH("Overlay Hooks"):::frontend
    end

    %% Client Browser Node
    BROWSER("Browser/Client (MSE)"):::frontend

    %% Developer & CI/CD Subgraph
    subgraph "Supporting Infrastructure"
        GH("GitHub Workflows"):::infrastructure
        BUILD("Build & Test Scripts"):::infrastructure
        LINT("Lint & TS Configs"):::infrastructure
    end

    %% Data Flow Connections
    MSP -->|"Processed Media"| BP
    MSP -->|"Processed Media"| PC
    BP -->|"Render Video"| BROWSER
    PC -->|"Render Video"| BROWSER
    OF ---|"Overlay Rendering"| BROWSER
    OL ---|"Overlay Rendering"| BROWSER
    OH ---|"Overlay Interactions"| BROWSER

    %% User Interaction Flow (dotted arrow)
    BROWSER -- "UserInteraction" --> BP
    BROWSER -- "UserInteraction" --> OF

    %% Click Events for Media Stream Pipelines
    click RTP "https://github.com/axiscommunications/media-stream-library-js/tree/main/src/streams/components/rtp"
    click RTSP "https://github.com/axiscommunications/media-stream-library-js/tree/main/src/streams/components/rtsp"
    click RTSP_AUTH "https://github.com/axiscommunications/media-stream-library-js/tree/main/src/streams/components/rtsp/auth"
    click MP4MUX "https://github.com/axiscommunications/media-stream-library-js/tree/main/src/streams/components/mp4-muxer"
    click MP4PAR "https://github.com/axiscommunications/media-stream-library-js/tree/main/src/streams/components/mp4-parser"
    click HTTP_MP4 "https://github.com/axiscommunications/media-stream-library-js/blob/main/src/streams/http-mp4-pipeline.ts"
    click RTSP_MP4 "https://github.com/axiscommunications/media-stream-library-js/blob/main/src/streams/rtsp-mp4-pipeline.ts"

    %% Click Events for Player Module
    click BP "https://github.com/axiscommunications/media-stream-library-js/blob/main/src/player/BasicPlayer.tsx"
    click PC "https://github.com/axiscommunications/media-stream-library-js/blob/main/src/player/Player.tsx"
    click EX_REACT "https://github.com/axiscommunications/media-stream-library-js/tree/main/example-player-react"
    click EX_WEB "https://github.com/axiscommunications/media-stream-library-js/tree/main/example-player-webcomponent"

    %% Click Events for Overlay Module
    click OF "https://github.com/axiscommunications/media-stream-library-js/blob/main/src/overlay/Foundation.tsx"
    click OL "https://github.com/axiscommunications/media-stream-library-js/blob/main/src/overlay/Liner.tsx"
    click OH "https://github.com/axiscommunications/media-stream-library-js/tree/main/src/overlay/hooks"

    %% Click Events for Supporting Infrastructure
    click GH "https://github.com/axiscommunications/media-stream-library-js/tree/main/.github/workflows"
    click BUILD "https://github.com/axiscommunications/media-stream-library-js/tree/main/justfile"
    click LINT "https://github.com/axiscommunications/media-stream-library-js/blob/main/tsconfig.json"

    %% Styling Classes
    classDef external fill:#F88,stroke:#333,stroke-width:2px;
    classDef backend fill:#BBF,stroke:#333,stroke-width:2px;
    classDef frontend fill:#BFB,stroke:#333,stroke-width:2px;
    classDef infrastructure fill:#FFB,stroke:#333,stroke-width:2px;
```
