How shuishen-cloud/blinderdetector works
灵眸伴途 is a visual-language-model travel assistant for people with visual impairments. Its principal workflow sends camera frames to backend perception, safety, navigation, and emergency logic; eligible announcements pass a gate and reach the user over a WebSocket for display, speech, and haptics. The frontend also supports destination voice input and route visualization, while a separate sender page simulates frame capture. The README documents optional cloud, map-routing, detector, and ASR integrations; several subsystem details are based on documentation rather than sampled implementation.
Main entry points: envelope.py Announcement envelope · net.js Client transport · main.js App assembly · main.py Application setup
Components 31
Frame capture
- Frame sender
web/js/sender.js
Backend transport
- Application setup
app/main.py - HTTP and WS routes
app/api/routes.py - Frame upload
app/api/uploads.py - ASR endpoint
app/api/speech.py - Announcement envelope
app/api/envelope.py - Broadcast hub
app/api/hub.py - Announcement gate
app/core/arbiter.py - Frame contracts
app/contracts.py - Runtime configuration
app/config.py
Assistance logic
- Scene perception
app/core/layers/perception.py - Safety analysis
app/core/layers/safety.py - Route planning
app/core/layers/navigation.py - Emergency states
app/core/layers/emergency.py - Fall detection
app/core/rules/fall.py - SOS escalation
app/core/rules/sos.py - Announcement phrasing
app/core/rules/phrasing.py
Providers and routing
- Implementation registry
app/core/registry.py - Vision providers
app/core/providers/openai_compat.py - Object detector
app/core/detectors/qwen_vl.py - Map router
app/core/routers/baidu.py - Speech recognizer
app/core/asr/dashscope.py
User experience
- Client transport
web/js/net.js - App assembly
web/js/main.js - Announcement UI
web/js/ui.js - Speech and haptics
web/js/speech.js - Destination input
web/js/nav.js - Voice input
web/js/voice.js - Route display
web/map.js
Other components
- User
- Browser client
Connections 43
- User → Frame sender: provides frames
- Frame sender → Client transport: uploads frames
web/js/sender.js - Client transport → Frame upload: POSTs frames
- Frame upload → Announcement envelope: builds response
app/api/routes.py - Application setup → HTTP and WS routes: builds routes
app/main.py - Application setup → Implementation registry: loads implementations
app/main.py - Application setup → Broadcast hub: creates hub
app/main.py - Application setup → Announcement envelope: creates envelope
app/main.py - Application setup → Announcement gate: creates gate
app/main.py - HTTP and WS routes → Frame upload: routes uploads
app/api/routes.py - HTTP and WS routes → ASR endpoint: routes audio
app/api/routes.py - HTTP and WS routes → Announcement envelope: routes requests
app/api/routes.py - HTTP and WS routes → Broadcast hub: serves stream
app/api/routes.py - Announcement envelope → Announcement gate: filters announcements
app/api/envelope.py - Announcement envelope → Broadcast hub: broadcasts announcements
app/api/envelope.py - Announcement envelope → Frame contracts: uses Frame
app/api/envelope.py - Announcement envelope → Scene perception: invokes layer
- Announcement envelope → Safety analysis: invokes layer
- Announcement envelope → Route planning: invokes layer
- Announcement envelope → Emergency states: invokes layer
- Scene perception → Vision providers: uses vision model
- Safety analysis → Object detector: detects obstacles
- Safety analysis → Announcement phrasing: phrases warnings
app/core/layers/safety.py - Route planning → Map router: requests routes
- Route planning → Announcement phrasing: phrases guidance
- Emergency states → Fall detection: tracks falls
- Emergency states → SOS escalation: advances SOS
- ASR endpoint → Implementation registry: selects recognizer
app/api/speech.py - ASR endpoint → Speech recognizer: transcribes audio
- Runtime configuration → Implementation registry: selects implementations
app/config.py - Browser client → Client transport: uses client transport
- Client transport → Broadcast hub: receives WS stream
- App assembly → Client transport: connects stream
web/js/main.js - App assembly → Announcement UI: initializes UI
web/js/main.js - App assembly → Destination input: initializes navigation
web/js/main.js - App assembly → Voice input: initializes voice
web/js/main.js - App assembly → Route display: initializes map
web/js/main.js - Client transport → Announcement UI: delivers announcements
web/js/main.js - Announcement UI → Speech and haptics: speaks announcements
web/js/ui.js - Voice input → Client transport: uploads audio
web/js/voice.js - Voice input → Destination input: submits destination
web/js/voice.js - Destination input → Client transport: requests route
web/js/nav.js - App assembly → Route display: forwards announcements
web/js/main.js