How microsoft/psi works
And Group Major Components
• Data Acquisition and Sensor Integration: Multiple folders cover sensor domains ─ “Sources/Audio”, “Sources/Imaging”, “Sources/Calibration”, “Sources/Devices”, “Sources/Kinect”, “Sources/RealSense”, “Sources/MixedReality”, “Sources/Speech”, etc. Each of these provides “acquisition” and “preprocessing” of data streams.
• Processing & Analytics Modules: Other folders in “Sources” (such as “Sources/Data”, “Sources/Filters”, “Sources/Language”, “Sources/Integrations”, “Sources/Onnx”, and “Sources/Spatial”) provide support for data transformations, machine learning (cognitive service wrappers, Onnx model runners), object detection, language interpretation, and spatial analytics.
• Application Layer: The “Applications” folder contains projects like SIGMA, the mixed‑reality task assistance prototype, and others that are built on top of the core Psi framework for specific use cases.
Layers and Data Flow
• At the bottom (or edge) are the sensor/input devices – cameras, microphones, Kinect, Azure Kinect, RealSense sensors, HoloLens, etc.
• Ingestion modules in their respective source areas read these sensor outputs and produce time‐synchronized data streams.
• The core Psi Pipeline Engine (the “Runtime” layer) is responsible for arranging these streams into processing graphs – the flow of data through operators, filters, and transformations (including synchronization, join, and fusion operators).
• Interprocess communication and remoting components (also part of the “Runtime” layer) allow distributed processing, streaming between processes, and data persistence (using PsiStore tools).
• The upper layers contain visualization, debugging & annotation tools (PsiStudio and other visualization modules) that subscribe to live or recorded streams.
• Finally, the Application layer (including Sigma and various samples) composes these capabilities into end-user applications.
Important Design Patterns and Architectural Facilities
• Component-Based and Extensible Design: Each sensor or processing domain is implemented as a module with clearly defined interfaces (often via NuGet packages) so that new components (or third-party integrations) can be “plugged” into a pipeline.
• Dataflow and Stream Processing: Central to the design is a pipeline model where data streams (with temporal semantics) are synchronized, transformed, and merged using well‑defined operators (as seen in “Sources/Runtime/Operators”).
• Layered Architecture: The separation between low-level device connectors, mid-level pipeline processing, and high-level visualization/applications is fundamental.
• Integration Points for AI / Cognitive Services: Modules in “Sources/Integrations” or “Sources/Onnx” clearly denote external service calls to, for example, Azure Cognitive Services.
• Sensor/Input Layer: Audio, Video, Depth, Kinect, MixedReality devices.
• Ingestion Modules: Domain-specific connectors and preprocessors (in Sources/Audio, Sources/Imaging, etc.).
• Core Runtime Layer: Pipeline engine, scheduling, stream operators, connectors, remoting, serialization (Sources/Runtime).
• Processing and Integration Layer: Filters, cognitive service connectors, Onnx model runners, spatial processing (Sources/Filters, Sources/Integrations, Sources/Onnx, Sources/Spatial).
• Persistence & Remoting: PsiStore components, network connectors, remoting modules.
• Visualization & Tooling: PsiStudio, visualization libraries (MSAGL, LiveCharts), diagnostic and debugging tools (Sources/Visualization and Sources/Tools).
• Application Layer: End-user applications (like Sigma in Applications) which orchestrate the above into actionable systems.