How joshpattani/resi_rendered-embodiment-of-social-interactions works
Main Components
• Sensors & Hardware Devices
– EEG (from OpenBCI Cyton boards)
– PPG (Photoplethysmography), GSR (Galvanic Skin Response), and temperature sensors
– Physical enclosures and 3D model designs (visible in the “3D Models” directory)
• Microcontrollers / Firmware
– MCU_code directory contains multiple submodules (e.g., “Communication”, “PlatformIO”, “Sensors”) with code for sensor data collection and wireless communication using Arduino, ESP32, and PlatformIO configurations
– Firmware projects include captive portals, OSC senders, and API testers used to connect sensors via WiFi and support both cloud connectivity and direct OSC transmission
• Data Acquisition and Preprocessing
– Brainwave streaming modules under “brainwaveStreaming” provide dual streaming setups, LSL (LabstreamingLayer) integration, and python scripts to receive EEG data
– An LSL-to-OSC bridge that converts real-time biosignal streams into control messages
• Signal Processing & Metrics Calculation
– Modules that process biometric inputs to extract metrics like band powers (Delta, Theta, Alpha, Beta), heart rate, and heart rate variability (HRV)
– This block maps raw sensor signals into descriptive metrics (the README gives ranges and intended interpretations)
• Audiovisual Mapping and Output
– OSC messages are routed to audiovisual rendering systems such as Max/MSP patches, Ableton Live projects, and TouchDesigner setups (referenced in “Max Patches” and the Ableton project folders)
– These components drive generative visuals (projected graphics) and spatial audio outputs (to speakers) based on biometric data
Relationships and Data Flow
• From MCU, data passes to the “Data Acquisition & Preprocessing” block. This block is responsible for collecting raw sensor outputs using tools like LabstreamingLayer and specialized MCU code (e.g., in brainwaveStreaming and PlatformIO folders).
• Split the OSC signals into two parallel paths heading to “Visuals” and “Audio”.
Architectural Patterns and Technologies
• Modular Separation: Each major subsystem (sensor acquisition, signal processing, OSC conversion, and audiovisual rendering) is clearly isolated, reflecting a component-based architecture.
• Real-Time Data Streaming: Emphasize the role of LabstreamingLayer (LSL) as the backbone for real-time signal exchange.
• Protocol and Messaging: Clearly annotate the use of OSC for control signaling between processing nodes and creative applications.
• Distributed Hardware/Software Integration: Note that the system bridges embedded hardware (MCU firmware) with desktop/laptop applications running Max/MSP, Ableton Live, and Python scripts.