How jphacks/kz_2504 works
Project Type and Purpose
• 4DX@HOME is a full-stack, multi-component experiential system combining:
a. AI-powered video analysis (Python + OpenCV + GPT-4o-mini)
b. Web application (React + TypeScript + Vite)
c. Backend API and real-time sync (FastAPI + WebSocket)
d. Embedded device hub (Raspberry Pi) and actuators (Arduino/ESP-12E)
File Structure Highlights
• Top-level directories:
• backend/ (FastAPI server, services, models, Docker setup)
• frontend/ (React/Vite app under 4dathome-app)
• hardware/ (device hub code, Arduino/ESP sketches, schematics)
• tools/sync-generator/ (AI video analysis scripts)
• docs/ (technical specifications)
• CI/Deployment artifacts: Dockerfiles, docker-compose.yml, DOCKER_SETUP.md
• API modules: device_registration, playback_control, video_management, preparation
• Services: continuous_sync_service (WebSocket loop), sync_data_service, video_service, preparation_service
Key Components
a. AI Video Analysis Engine
• Input: raw MP4 → analyze_video.py + generator.py → JSON timeline
• External dependency: OpenAI Vision API (GPT-4o-mini) and OpenCV
b. Backend Server
• REST endpoints (FastAPI) for session, video management, device registration
• WebSocket endpoint for real-time timestamp sync
• Services layer orchestrates timeline dispatch to clients
c. Frontend Web App
• SPA (React + TypeScript)
• Uses REST to create/join sessions, retrieve timeline JSON
• WebSocket client for real-time sync messages → triggers UI & local clock
d. Device Hub (Raspberry Pi)
• WebSocket and/or TCP client to backend
• hardware_server.py: receives sync messages → maintains timeline queue
• serial_controller.py + paho-mqtt: routes commands to Arduino & ESP-12E
e. Actuators
• Arduino Uno (light, wind, water) via serial
• ESP-12E (vibration motor) via MQTT
• Controlled in real time according to timeline JSON
Component Interactions
• (1) User navigates web app → (2) REST API device_registration/session create
• (3) Video uploaded/selected → (4) Backend video_service calls AI Engine → JSON stored
• (5) Frontend fetches JSON → (6) Frontend opens WebSocket → (7) Backend continuous_sync_service pushes timestamps
• (8) Raspberry Pi hub subscribed to same WebSocket → (9) hardware_server queues events → (10) serial & MQTT to actuators
Architectural Patterns & Design Principles
• Layered architecture in backend (API → services → models)
• Single-page application (React Router DOM) separates views/pages
• Hybrid comms: REST for control, WebSocket for real-time sync, serial/MQTT for hardware
• Modular hardware integration: device hub abstracts actuator control