How darrenthiores/handtalk works
Project type and purpose
• This is a SwiftUI-based iOS App Playground (“HandTalk”) that uses Core ML for real-time hand-pose detection and a quiz/learning mode.
• Purpose:
• Feature A (“Detection”): capture live camera frames, run a hand-sign ML model, display and speak the recognized letter/word.
• Feature B (“Learning”): present randomized hand-sign prompts, capture user poses, verify via ML, show correct pose on failure, advance on success.
File structure
• Top-level SwiftPM and Xcode workspace files (playground)
• Modules by feature:
• Boarding (onboarding screens)
• Detection (real-time detection view + controller + overlay)
• Learning (quiz flow, ViewModel, state machine, custom camera component)
• Main (home view, navigation components)
• Shared folders:
• Components/Camera (reusable CameraPreview)
• Model (Classifier protocols, HandPoseClassifier, HandSignReader wrapper)
• MLModel (.mlmodel resources)
• Entry point: MyApp.swift, Package.swift
README insights
• ML model trained with CreateML (ASL dataset)
• App Playground specifics: MLModel must be copied in Package.swift
• Two main use cases: translation and learning
a. Main components
• UI Layer (SwiftUI Views)
– BoardingView
– DetectionView + DetectionCameraViewController
– LearningView + sub-components + LearningViewModel
– MainView (navigation hub)
• Camera Layer
– AVFoundation wrapper (CameraManager, CameraDelegate)
– CameraPreview (reusable SwiftUI UIViewRepresentable)
• ML Inference Layer
– HandSignReader.mlmodel
– Model APIs (Classifier, HandPoseClassifier)
• Business Logic / State Management
– LearningViewModel (MVVM)
– DetectionController (imperative handling)
• Services
– Text-to-Speech synthesizer (system API invoked in Detection)
• Assets & Configuration
b. Relationships & data flows
1. User interacts with SwiftUI View → triggers Camera capture.
2. CameraManager streams CMSampleBuffer → forwarded to Model API.
3. Core ML model inference → returns HandSign classification.
4. Result flows back to ViewModel / Controller.
5. UI displays classification and invokes TTS service.
6. In Learning mode: ViewModel implements quiz state machine → issues new prompt → repeats capture/inference cycle → updates score & displays correct pose if wrong.
c. Architectural patterns
• MVVM in Learning: View ↔ ViewModel ↔ Model
• Reusable components (CameraPreview, CaptureButton, PredictionAlert)
• Separation of concerns: UI vs. Camera vs. ML vs. Theme/Assets
d. Key technologies
• SwiftUI
• Combine or @State / @Published (for reactive binding)
• AVFoundation (camera capture)
• Core ML (.mlmodel + CreateML)
• Swift Package Manager (playground packaging)
• iOS Text-to-Speech (AVSpeechSynthesizer)