# microsoft/psi architecture

> Architecture diagram of the GitHub repository microsoft/psi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/psi
- Repository: https://github.com/microsoft/psi

## Overview

##### And Group Major Components
- Data Acquisition and Sensor Integration: Multiple folders cover sensor domains ─ “Sources/Audio”, “Sources/Imaging”, “Sources/Calibration”, “Sources/Devices”, “Sources/Kinect”, “Sources/RealSense”, “Sources/MixedReality”, “Sources/Speech”, etc. Each of these provides “acquisition” and “preprocessing” of data streams.
- Processing & Analytics Modules: Other folders in “Sources” (such as “Sources/Data”, “Sources/Filters”, “Sources/Language”, “Sources/Integrations”, “Sources/Onnx”, and “Sources/Spatial”) provide support for data transformations, machine learning (cognitive service wrappers, Onnx model runners), object detection, language interpretation, and spatial analytics.
- Application Layer: The “Applications” folder contains projects like SIGMA, the mixed‑reality task assistance prototype, and others that are built on top of the core Psi framework for specific use cases.

##### Layers and Data Flow
- At the bottom (or edge) are the sensor/input devices – cameras, microphones, Kinect, Azure Kinect, RealSense sensors, HoloLens, etc.
- Ingestion modules in their respective source areas read these sensor outputs and produce time‐synchronized data streams.
- The core Psi Pipeline Engine (the “Runtime” layer) is responsible for arranging these streams into processing graphs – the flow of data through operators, filters, and transformations (including synchronization, join, and fusion operators).
- Interprocess communication and remoting components (also part of the “Runtime” layer) allow distributed processing, streaming between processes, and data persistence (using PsiStore tools).
- The upper layers contain visualization, debugging & annotation tools (PsiStudio and other visualization modules) that subscribe to live or recorded streams.
- Finally, the Application layer (including Sigma and various samples) composes these capabilities into end-user applications.

##### Important Design Patterns and Architectural Facilities
- Component-Based and Extensible Design: Each sensor or processing domain is implemented as a module with clearly defined interfaces (often via NuGet packages) so that new components (or third-party integrations) can be “plugged” into a pipeline.
- Dataflow and Stream Processing: Central to the design is a pipeline model where data streams (with temporal semantics) are synchronized, transformed, and merged using well‑defined operators (as seen in “Sources/Runtime/Operators”).
- Layered Architecture: The separation between low-level device connectors, mid-level pipeline processing, and high-level visualization/applications is fundamental.
- Integration Points for AI / Cognitive Services: Modules in “Sources/Integrations” or “Sources/Onnx” clearly denote external service calls to, for example, Azure Cognitive Services.

- Sensor/Input Layer: Audio, Video, Depth, Kinect, MixedReality devices.
- Ingestion Modules: Domain-specific connectors and preprocessors (in Sources/Audio, Sources/Imaging, etc.).
- Core Runtime Layer: Pipeline engine, scheduling, stream operators, connectors, remoting, serialization (Sources/Runtime).
- Processing and Integration Layer: Filters, cognitive service connectors, Onnx model runners, spatial processing (Sources/Filters, Sources/Integrations, Sources/Onnx, Sources/Spatial).
- Persistence & Remoting: PsiStore components, network connectors, remoting modules.
- Visualization & Tooling: PsiStudio, visualization libraries (MSAGL, LiveCharts), diagnostic and debugging tools (Sources/Visualization and Sources/Tools).
- Application Layer: End-user applications (like Sigma in Applications) which orchestrate the above into actionable systems.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Sensor/Input Layer
    subgraph "Sensor/Input Layer"
        Audio["Audio Acquisition"]:::sensor
        Imaging["Imaging Acquisition"]:::sensor
        Calibration["Calibration Modules"]:::sensor
        Devices["Devices Integration"]:::sensor
        Kinect["Kinect Integration"]:::sensor
        RealSense["RealSense Integration"]:::sensor
        MixedReality["MixedReality Integration"]:::sensor
        Speech["Speech Integration"]:::sensor
    end

    %% Core Runtime Layer
    subgraph "Core Runtime Layer"
        CoreRuntime["Core Runtime & Pipeline Engine"]:::runtime
    end

    %% Processing & Analytics Layer
    subgraph "Processing & Analytics Layer"
        DataProcessing["Data Processing"]:::processing
        Filters["Filters"]:::processing
        Language["Language Processing"]:::processing
        Integrations["Integrations"]:::processing
        Onnx["Onnx Execution"]:::processing
        Spatial["Spatial Analytics"]:::processing
    end

    %% Visualization & Tools Layer
    subgraph "Visualization & Tools Layer"
        Visualization["Visualization"]:::visualization
        Tools["Tools"]:::visualization
        Toolkits["Toolkits"]:::visualization
    end

    %% Application Layer
    subgraph "Application Layer"
        Applications["Applications"]:::application
    end

    %% Dataflow from Sensor/Input Layer to Core Runtime Layer
    Audio -->|"ingests"| CoreRuntime
    Imaging -->|"ingests"| CoreRuntime
    Calibration -->|"ingests"| CoreRuntime
    Devices -->|"ingests"| CoreRuntime
    Kinect -->|"ingests"| CoreRuntime
    RealSense -->|"ingests"| CoreRuntime
    MixedReality -->|"ingests"| CoreRuntime
    Speech -->|"ingests"| CoreRuntime

    %% Dataflow from Core Runtime Layer to Processing & Analytics Layer
    CoreRuntime -->|"processes"| DataProcessing
    CoreRuntime -->|"processes"| Filters
    CoreRuntime -->|"processes"| Language
    CoreRuntime -->|"processes"| Integrations
    CoreRuntime -->|"processes"| Onnx
    CoreRuntime -->|"processes"| Spatial

    %% Dataflow from Processing & Analytics Layer to Visualization & Tools Layer
    DataProcessing -->|"sends"| Visualization
    Filters -->|"sends"| Visualization
    Language -->|"sends"| Visualization
    Integrations -->|"sends"| Visualization
    Onnx -->|"sends"| Visualization
    Spatial -->|"sends"| Visualization

    %% Dataflow from Visualization & Tools Layer to Application Layer
    Visualization -->|"composes"| Applications
    Tools -->|"composes"| Applications
    Toolkits -->|"composes"| Applications

    %% Click Events
    click CoreRuntime "https://github.com/microsoft/psi/tree/master/Sources/Runtime"
    click Audio "https://github.com/microsoft/psi/tree/master/Sources/Audio"
    click Imaging "https://github.com/microsoft/psi/tree/master/Sources/Imaging"
    click Calibration "https://github.com/microsoft/psi/tree/master/Sources/Calibration"
    click Devices "https://github.com/microsoft/psi/tree/master/Sources/Devices"
    click Kinect "https://github.com/microsoft/psi/tree/master/Sources/Kinect"
    click RealSense "https://github.com/microsoft/psi/tree/master/Sources/RealSense"
    click MixedReality "https://github.com/microsoft/psi/tree/master/Sources/MixedReality"
    click Speech "https://github.com/microsoft/psi/tree/master/Sources/Speech"
    click DataProcessing "https://github.com/microsoft/psi/tree/master/Sources/Data"
    click Filters "https://github.com/microsoft/psi/tree/master/Sources/Filters"
    click Language "https://github.com/microsoft/psi/tree/master/Sources/Language"
    click Integrations "https://github.com/microsoft/psi/tree/master/Sources/Integrations"
    click Onnx "https://github.com/microsoft/psi/tree/master/Sources/Integrations/Onnx"
    click Spatial "https://github.com/microsoft/psi/tree/master/Sources/Spatial"
    click Visualization "https://github.com/microsoft/psi/tree/master/Sources/Visualization"
    click Tools "https://github.com/microsoft/psi/tree/master/Sources/Tools"
    click Toolkits "https://github.com/microsoft/psi/tree/master/Sources/Toolkits"
    click Applications "https://github.com/microsoft/psi/tree/master/Applications"

    %% Styles
    classDef sensor fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef runtime fill:#cce5ff,stroke:#003399,stroke-width:2px;
    classDef processing fill:#ffcc99,stroke:#cc6600,stroke-width:2px;
    classDef visualization fill:#ffd699,stroke:#ff8000,stroke-width:2px;
    classDef application fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
```
