# jphacks/kz_2504 architecture

> Architecture diagram of the GitHub repository jphacks/kz_2504, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jphacks/kz_2504
- Repository: https://github.com/jphacks/kz_2504

## Overview

##### Project Type and Purpose
- 4DX@HOME is a full-stack, multi-component experiential system combining:
a. AI-powered video analysis (Python + OpenCV + GPT-4o-mini)
b. Web application (React + TypeScript + Vite)
c. Backend API and real-time sync (FastAPI + WebSocket)
d. Embedded device hub (Raspberry Pi) and actuators (Arduino/ESP-12E)

##### File Structure Highlights
- Top-level directories:
- backend/ (FastAPI server, services, models, Docker setup)
- frontend/ (React/Vite app under 4dathome-app)
- hardware/ (device hub code, Arduino/ESP sketches, schematics)
- tools/sync-generator/ (AI video analysis scripts)
- docs/ (technical specifications)
- CI/Deployment artifacts: Dockerfiles, docker-compose.yml, DOCKER_SETUP.md
- API modules: device_registration, playback_control, video_management, preparation
- Services: continuous_sync_service (WebSocket loop), sync_data_service, video_service, preparation_service

##### Key Components
a. AI Video Analysis Engine
- Input: raw MP4 → analyze_video.py + generator.py → JSON timeline
- External dependency: OpenAI Vision API (GPT-4o-mini) and OpenCV
b. Backend Server
- REST endpoints (FastAPI) for session, video management, device registration
- WebSocket endpoint for real-time timestamp sync
- Services layer orchestrates timeline dispatch to clients
c. Frontend Web App
- SPA (React + TypeScript)
- Uses REST to create/join sessions, retrieve timeline JSON
- WebSocket client for real-time sync messages → triggers UI & local clock
d. Device Hub (Raspberry Pi)
- WebSocket and/or TCP client to backend
- hardware_server.py: receives sync messages → maintains timeline queue
- serial_controller.py + paho-mqtt: routes commands to Arduino & ESP-12E
e. Actuators
- Arduino Uno (light, wind, water) via serial
- ESP-12E (vibration motor) via MQTT
- Controlled in real time according to timeline JSON

##### Component Interactions
- (1) User navigates web app → (2) REST API device_registration/session create
- (3) Video uploaded/selected → (4) Backend video_service calls AI Engine → JSON stored
- (5) Frontend fetches JSON → (6) Frontend opens WebSocket → (7) Backend continuous_sync_service pushes timestamps
- (8) Raspberry Pi hub subscribed to same WebSocket → (9) hardware_server queues events → (10) serial & MQTT to actuators

##### Architectural Patterns & Design Principles
- Layered architecture in backend (API → services → models)
- Single-page application (React Router DOM) separates views/pages
- Hybrid comms: REST for control, WebSocket for real-time sync, serial/MQTT for hardware
- Modular hardware integration: device hub abstracts actuator control

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "AI Video Analysis Engine" 
        A1["analyze_video.py"]:::service
        A2["generator.py"]:::service
        A3["playback_video.py"]:::service
        A4["requirements.txt"]:::service
    end
    subgraph "Backend Server (FastAPI)" 
        B1["main.py"]:::service
        subgraph "API Modules"
            B2["device_registration.py"]:::service
            B3["video_management.py"]:::service
            B4["playback_control.py"]:::service
            B5["preparation.py"]:::service
        end
        subgraph "Services Layer"
            B6["video_service.py"]:::service
            B7["continuous_sync_service.py"]:::service
            B8["sync_data_service.py"]:::service
            B9["preparation_service.py"]:::service
        end
        subgraph "Models"
            B10["video.py"]:::service
            B11["device.py"]:::service
            B12["playback.py"]:::service
            B13["preparation.py"]:::service
        end
        subgraph "Config/Deployment"
            B14["settings.py"]:::service
            B15["Dockerfile"]:::service
            B16["docker-compose.yml"]:::service
            B17["DOCKER_SETUP.md"]:::service
        end
    end
    subgraph "Frontend Web App (React + TS)" 
        F1["App.tsx"]:::frontend
        F2["main.tsx"]:::frontend
        F3["pages/*"]:::frontend
        F4["components/*"]:::frontend
        F5["useSessionApi.ts"]:::frontend
        F6["vite.config.tsx"]:::frontend
        F7["socket.ts"]:::frontend
        F8["useWebSocket.ts"]:::frontend
    end
    subgraph "Device Hub (Raspberry Pi)" 
        H1["hub.py"]:::hardware
        H2["serial_controller.py"]:::hardware
        H3["websocket_client.py"]:::hardware
        H4["config.json"]:::hardware
        H5["requirements.txt"]:::hardware
        H6["4dx-hub.service"]:::hardware
    end
    subgraph "Actuators" 
        S["vibration-motor/actuator.ino"]:::actuator
        D["scent-diffuser/README.md"]:::actuator
        L["lights.ino"]:::actuator
        V["vibration.ino"]:::actuator
        W["water.ino"]:::actuator
    end
    subgraph "External Services"
        OpenAI["OpenAI Vision API"]:::external
    end

    OpenAI -->|HTTPS/JSON| A1
    A1 -->|"calls generator"| A2
    A1 -->|"uses playback_video"| A3
    A3 -->|"outputs JSON Timeline"| B6
    F1 -->|HTTPS/JSON| B1
    F1 -->|WSS/JSON| B7
    H1 -->|WSS/JSON| B7
    B7 -->|WSS/JSON| F1
    B7 -->|WSS/JSON| H1
    H1 -->|Serial/MQTT| S
    H1 -->|Serial/MQTT| D
    H1 -->|Serial/MQTT| L
    H1 -->|Serial/MQTT| V
    H1 -->|Serial/MQTT| W

    click A1 "https://github.com/jphacks/kz_2504/blob/main/tools/sync-generator/analyze_video.py"
    click A2 "https://github.com/jphacks/kz_2504/blob/main/tools/sync-generator/generator.py"
    click A3 "https://github.com/jphacks/kz_2504/blob/main/tools/sync-generator/playback_video.py"
    click A4 "https://github.com/jphacks/kz_2504/blob/main/tools/sync-generator/requirements.txt"
    click B1 "https://github.com/jphacks/kz_2504/blob/main/backend/app/main.py"
    click B2 "https://github.com/jphacks/kz_2504/blob/main/backend/app/api/device_registration.py"
    click B3 "https://github.com/jphacks/kz_2504/blob/main/backend/app/api/video_management.py"
    click B4 "https://github.com/jphacks/kz_2504/blob/main/backend/app/api/playback_control.py"
    click B5 "https://github.com/jphacks/kz_2504/blob/main/backend/app/api/preparation.py"
    click B6 "https://github.com/jphacks/kz_2504/blob/main/backend/app/services/video_service.py"
    click B7 "https://github.com/jphacks/kz_2504/blob/main/backend/app/services/continuous_sync_service.py"
    click B8 "https://github.com/jphacks/kz_2504/blob/main/backend/app/services/sync_data_service.py"
    click B9 "https://github.com/jphacks/kz_2504/blob/main/backend/app/services/preparation_service.py"
    click B10 "https://github.com/jphacks/kz_2504/blob/main/backend/app/models/video.py"
    click B11 "https://github.com/jphacks/kz_2504/blob/main/backend/app/models/device.py"
    click B12 "https://github.com/jphacks/kz_2504/blob/main/backend/app/models/playback.py"
    click B13 "https://github.com/jphacks/kz_2504/blob/main/backend/app/models/preparation.py"
    click B14 "https://github.com/jphacks/kz_2504/blob/main/backend/app/config/settings.py"
    click B15 "https://github.com/jphacks/kz_2504/tree/main/backend/Dockerfile"
    click B16 "https://github.com/jphacks/kz_2504/blob/main/backend/docker-compose.yml"
    click B17 "https://github.com/jphacks/kz_2504/blob/main/backend/DOCKER_SETUP.md"
    click F1 "https://github.com/jphacks/kz_2504/blob/main/frontend/4dathome-app/src/App.tsx"
    click F2 "https://github.com/jphacks/kz_2504/blob/main/frontend/4dathome-app/src/main.tsx"
    click F3 "https://github.com/jphacks/kz_2504/tree/main/frontend/4dathome-app/src/pages/*"
    click F4 "https://github.com/jphacks/kz_2504/tree/main/frontend/4dathome-app/src/components/*"
    click F5 "https://github.com/jphacks/kz_2504/blob/main/frontend/4dathome-app/src/hooks/useSessionApi.ts"
    click F6 "https://github.com/jphacks/kz_2504/blob/main/frontend/4dathome-app/vite.config.tsx"
    click F7 "https://github.com/jphacks/kz_2504/blob/main/frontend/4dathome-app/src/utils/socket.ts"
    click F8 "https://github.com/jphacks/kz_2504/blob/main/frontend/4dathome-app/src/hooks/useWebSocket.ts"
    click H1 "https://github.com/jphacks/kz_2504/blob/main/hardware/device-hub/src/hub.py"
    click H2 "https://github.com/jphacks/kz_2504/blob/main/hardware/device-hub/src/serial_controller.py"
    click H3 "https://github.com/jphacks/kz_2504/blob/main/hardware/device-hub/src/websocket_client.py"
    click H4 "https://github.com/jphacks/kz_2504/blob/main/hardware/device-hub/config/config.json"
    click H5 "https://github.com/jphacks/kz_2504/blob/main/hardware/device-hub/requirements.txt"
    click H6 "https://github.com/jphacks/kz_2504/blob/main/hardware/device-hub/systemd/4dx-hub.service"
    click S "https://github.com/jphacks/kz_2504/blob/main/hardware/actuators/vibration-motor/actuator.ino"
    click D "https://github.com/jphacks/kz_2504/blob/main/hardware/actuators/scent-diffuser/README.md"
    click L "https://github.com/jphacks/kz_2504/blob/main/hardware/lights.ino"
    click V "https://github.com/jphacks/kz_2504/blob/main/hardware/vibration.ino"
    click W "https://github.com/jphacks/kz_2504/blob/main/hardware/water.ino"

    classDef service fill:#cce5ff,stroke:#66b0ff,stroke-width:1px
    classDef frontend fill:#e6ffe6,stroke:#47b147,stroke-width:1px
    classDef hardware fill:#ffe6cc,stroke:#ff9947,stroke-width:1px
    classDef actuator fill:#ffffcc,stroke:#ffcc00,stroke-width:1px
    classDef external fill:#f2f2f2,stroke:#999999,stroke-width:1px
```
