# darrenthiores/handtalk architecture

> Architecture diagram of the GitHub repository darrenthiores/handtalk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darrenthiores/handtalk
- Repository: https://github.com/darrenthiores/handtalk

## Overview

##### Project type and purpose
- This is a SwiftUI-based iOS App Playground (“HandTalk”) that uses Core ML for real-time hand-pose detection and a quiz/learning mode.
- Purpose:
- Feature A (“Detection”): capture live camera frames, run a hand-sign ML model, display and speak the recognized letter/word.
- Feature B (“Learning”): present randomized hand-sign prompts, capture user poses, verify via ML, show correct pose on failure, advance on success.

##### File structure
- Top-level SwiftPM and Xcode workspace files (playground)
- Modules by feature:
- Boarding (onboarding screens)
- Detection (real-time detection view + controller + overlay)
- Learning (quiz flow, ViewModel, state machine, custom camera component)
- Main (home view, navigation components)
- Shared folders:
- Components/Camera (reusable CameraPreview)
- Model (Classifier protocols, HandPoseClassifier, HandSignReader wrapper)
- MLModel (.mlmodel resources)
- Entry point: MyApp.swift, Package.swift

##### README insights
- ML model trained with CreateML (ASL dataset)
- App Playground specifics: MLModel must be copied in Package.swift
- Two main use cases: translation and learning

a. Main components
- UI Layer (SwiftUI Views)
– BoardingView
– DetectionView + DetectionCameraViewController
– LearningView + sub-components + LearningViewModel
– MainView (navigation hub)
- Camera Layer
– AVFoundation wrapper (CameraManager, CameraDelegate)
– CameraPreview (reusable SwiftUI UIViewRepresentable)
- ML Inference Layer
– HandSignReader.mlmodel
– Model APIs (Classifier, HandPoseClassifier)
- Business Logic / State Management
– LearningViewModel (MVVM)
– DetectionController (imperative handling)
- Services
– Text-to-Speech synthesizer (system API invoked in Detection)
- Assets & Configuration

b. Relationships & data flows
1. User interacts with SwiftUI View → triggers Camera capture.
2. CameraManager streams CMSampleBuffer → forwarded to Model API.
3. Core ML model inference → returns HandSign classification.
4. Result flows back to ViewModel / Controller.
5. UI displays classification and invokes TTS service.
6. In Learning mode: ViewModel implements quiz state machine → issues new prompt → repeats capture/inference cycle → updates score & displays correct pose if wrong.

c. Architectural patterns
- MVVM in Learning: View ↔ ViewModel ↔ Model
- Reusable components (CameraPreview, CaptureButton, PredictionAlert)
- Separation of concerns: UI vs. Camera vs. ML vs. Theme/Assets

d. Key technologies
- SwiftUI
- Combine or @State / @Published (for reactive binding)
- AVFoundation (camera capture)
- Core ML (.mlmodel + CreateML)
- Swift Package Manager (playground packaging)
- iOS Text-to-Speech (AVSpeechSynthesizer)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% UI Layer
    subgraph "UI Layer"
        subgraph Boarding
            BV["BoardingView"]:::ui
            BSV["BoardingSectionView"]:::ui
        end
        subgraph Main
            MV["MainView"]:::ui
            HSV["HandSignView"]:::ui
            LC["LearnCard"]:::ui
        end
        subgraph Detection UI
            DV["DetectionView"]:::ui
        end
        subgraph Learning UI
            LV["LearningView"]:::ui
            CB["CaptureButton"]:::ui
            PA["PredictionAlert"]:::ui
            SB["SwitchButton"]:::ui
        end
    end

    %% Camera Layer
    subgraph "Camera Layer"
        PREV["CameraPreview"]:::camera
        DCV["DetectionCameraView"]:::camera
        DCVC["DetectionCameraViewController"]:::camera
        CM["CameraManager"]:::camera
        CD["CameraDelegate"]:::camera
        LCV["LearnCameraView"]:::camera
    end

    %% ML Inference Layer
    subgraph "ML Inference Layer"
        CPROTO["Classifier"]:::ml
        HPC["HandPoseClassifier"]:::ml
        HSR["HandSignReader"]:::ml
        HSRM["HandSignReader.mlmodel"]:::ml
        HS["HandSign"]:::ml
    end

    %% Business Logic
    subgraph "Business Logic"
        LVM["LearningViewModel"]:::logic
        LS["LearningState"]:::logic
        LE["LearningEvent"]:::logic
        DC["DetectionController"]:::logic
    end

    %% Services
    subgraph "Services"
        TTS["AVSpeechSynthesizer"]:::service
    end

    %% Assets & Theme
    subgraph "Assets & Theme"
        ASSETS["Assets.xcassets"]:::asset
        THEME["ThemeExtension"]:::asset
    end

    %% Flows
    BV -->|"complete"| MV
    MV -->|"select Detection"| DV
    MV -->|"select Learning"| LV

    DV --> DCV
    DCV --> PREV
    DCV --> DCVC
    DCVC -->|capture frame| CPROTO
    CPROTO -->|prediction| DCVC
    DCVC -->|result| DV
    DCVC -->|speak| TTS

    LV --> LVM
    LV --> LCV
    LVM -->|bind| LV
    LCV --> PREV
    LVM --> CM
    CM --> CD
    CD -->|capture frame| CPROTO
    CPROTO -->|prediction| CD
    CD --> LVM
    LVM --> LV

    %% Shared ML components
    CPROTO --> HPC
    HPC --> HSR
    HSR --> HSRM
    HSR --> HS

    %% Assets & Theme usage
    ASSETS --> BV
    ASSETS --> MV
    ASSETS --> DV
    ASSETS --> LV
    THEME --> BV
    THEME --> MV
    THEME --> DV
    THEME --> LV

    %% Legend
    classDef ui fill:#D0E8FF,stroke:#0366d6
    classDef camera fill:#FFE5B4,stroke:#d2691e
    classDef ml fill:#D4EDDA,stroke:#28a745
    classDef logic fill:#FFF3CD,stroke:#856404
    classDef service fill:#E2D6F3,stroke:#6f42c1
    classDef asset fill:#E0E0E0,stroke:#6c757d

    %% Click events
    click BV "https://github.com/darrenthiores/handtalk/blob/main/Boarding/BoardingView.swift"
    click BSV "https://github.com/darrenthiores/handtalk/blob/main/Boarding/Components/BoardingSectionView.swift"
    click MV "https://github.com/darrenthiores/handtalk/blob/main/Main/MainView.swift"
    click HSV "https://github.com/darrenthiores/handtalk/blob/main/Main/Components/HandSignView.swift"
    click LC "https://github.com/darrenthiores/handtalk/blob/main/Main/Components/LearnCard.swift"
    click DV "https://github.com/darrenthiores/handtalk/blob/main/Detection/DetectionView.swift"
    click DCV "https://github.com/darrenthiores/handtalk/blob/main/Detection/Camera/DetectionCameraView.swift"
    click DCVC "https://github.com/darrenthiores/handtalk/blob/main/Detection/Camera/DetectionCameraViewController.swift"
    click PREV "https://github.com/darrenthiores/handtalk/blob/main/Components/Camera/CameraPreview.swift"
    click CM "https://github.com/darrenthiores/handtalk/blob/main/Learning/Camera/CameraManager.swift"
    click CD "https://github.com/darrenthiores/handtalk/blob/main/Learning/Camera/CameraDelegate.swift"
    click LCV "https://github.com/darrenthiores/handtalk/blob/main/Learning/Camera/LearnCameraView.swift"
    click LV "https://github.com/darrenthiores/handtalk/blob/main/Learning/LearningView.swift"
    click CB "https://github.com/darrenthiores/handtalk/blob/main/Learning/Components/CaptureButton.swift"
    click PA "https://github.com/darrenthiores/handtalk/blob/main/Learning/Components/PredictionAlert.swift"
    click SB "https://github.com/darrenthiores/handtalk/blob/main/Learning/Components/SwitchButton.swift"
    click LVM "https://github.com/darrenthiores/handtalk/blob/main/Learning/LearningViewModel.swift"
    click LS "https://github.com/darrenthiores/handtalk/blob/main/Learning/LearningState.swift"
    click LE "https://github.com/darrenthiores/handtalk/blob/main/Learning/LearningEvent.swift"
    click CPROTO "https://github.com/darrenthiores/handtalk/blob/main/Model/Classifier.swift"
    click HPC "https://github.com/darrenthiores/handtalk/blob/main/Model/HandPoseClassifier.swift"
    click HSR "https://github.com/darrenthiores/handtalk/blob/main/Model/HandSignReader.swift"
    click HSRM "https://github.com/darrenthiores/handtalk/blob/main/MLModel/HandSignReader.mlmodel"
    click HS "https://github.com/darrenthiores/handtalk/blob/main/Model/HandSign.swift"
    click THEME "https://github.com/darrenthiores/handtalk/blob/main/Theme/ColorExtension.swift"
    click ASSETS "https://github.com/darrenthiores/handtalk/blob/main/Assets.xcassets"
```
