# troyhow0413/avsd_ger architecture

> Architecture diagram of the GitHub repository troyhow0413/avsd_ger, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/troyhow0413/avsd_ger
- Repository: https://github.com/troyhow0413/avsd_ger

## Overview

AVSD-GER performs identity-conditioned, closed-loop correction of audio-visual speech recognition for multi-speaker sessions. Its principal workflow processes ordered session turns through a per-turn pipeline: prepare visual mouth input when needed, obtain audio and visual hypotheses, retrieve an enrolled speaker identity, align ASR features using identity, generate a corrected transcript, and apply confidence-gated feedback that may update the identity pool. The graph also preserves the documented enrollment and evaluation roles. Pipeline and session excerpts are partial, so links between stages follow their explicit module descriptions and documented design; uncertain internal call details are omitted.

Main entry points: [`avsd_ger/pipeline.py`](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/pipeline.py) (AVSD-GER pipeline), [`one_go/c1_enroll.py`](https://github.com/troyhow0413/avsd_ger/blob/main/one_go/c1_enroll.py) (Speaker enrollment), [`avsd_ger/eval/session.py`](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/eval/session.py) (Turn runner), [`avsd_ger/c1_identity/identity_pool.py`](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/identity_pool.py) (Identity pool)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_session["Session workflow"]
  node_session_runner["Turn runner<br/>[session.py]"]
  node_pipeline["AVSD-GER pipeline<br/>[pipeline.py]"]
  node_metrics["Evaluation metrics<br/>[metrics.py]"]
end

subgraph group_identity["C1 Identity"]
  node_voice_encoder["Voice encoder<br/>[voice_encoder.py]"]
  node_face_encoder["Face encoder<br/>[face_encoder.py]"]
  node_identity_pool[("Identity pool<br/>[identity_pool.py]")]
  node_identity_gate["Identity gate<br/>[gate.py]"]
  node_enrollment["Speaker enrollment<br/>[c1_enroll.py]"]
end

subgraph group_recognition["Recognition and C2"]
  node_mouth_roi["Mouth ROI<br/>[mouth_roi.py]"]
  node_frontend_registry["Frontend registry<br/>[registry.py]"]
  node_whisper["Whisper ASR<br/>[asr_whisper.py]"]
  node_vsr["AV-HuBERT VSR<br/>[vsr_avhubert.py]"]
  node_aligner["ID-conditioned aligner"]
  node_prompt_builder["Prompt builder<br/>[prompt_builder.py]"]
  node_ger_head["GER generator<br/>[ger_head.py]"]
  node_soft_bridge["Soft-token bridge"]
  node_ctc_loss["CTC training head<br/>[ctc_loss.py]"]
end

subgraph group_feedback["C3 Feedback"]
  node_confidence["Confidence scorer"]
  node_closed_loop["Feedback loop<br/>[closed_loop.py]"]
  node_safety["GER safety gate<br/>[ger_safety.py]"]
end

node_user(("Session input"))

node_user -->|"provides turns"| node_session_runner
node_session_runner -->|"processes each turn"| node_pipeline
node_pipeline -->|"extracts mouth input"| node_mouth_roi
node_pipeline -->|"selects frontend"| node_frontend_registry
node_pipeline -->|"recognizes audio"| node_whisper
node_pipeline -->|"recognizes visual speech"| node_vsr
node_pipeline -->|"encodes voice"| node_voice_encoder
node_pipeline -->|"encodes face"| node_face_encoder
node_pipeline -->|"queries identity"| node_identity_pool
node_pipeline -->|"gates identity"| node_identity_gate
node_enrollment -->|"encodes enrollment"| node_voice_encoder
node_enrollment -->|"encodes enrollment"| node_face_encoder
node_enrollment -->|"enrolls speaker"| node_identity_pool
node_pipeline -->|"aligns ASR features"| node_aligner
node_pipeline -->|"builds correction prompt"| node_prompt_builder
node_pipeline -->|"generates correction"| node_ger_head
node_pipeline -->|"bridges aligned features"| node_soft_bridge
node_pipeline -->|"scores hypotheses"| node_confidence
node_pipeline -->|"checks feedback action"| node_closed_loop
node_pipeline -->|"checks generated text"| node_safety
node_closed_loop -->|"requests pool update"| node_identity_pool
node_session_runner -->|"evaluates results"| node_metrics
node_ctc_loss -.->|"trains aligned features"| node_aligner

click node_session_runner "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/eval/session.py"
click node_pipeline "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/pipeline.py"
click node_mouth_roi "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/frontend/mouth_roi.py"
click node_frontend_registry "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/frontend/registry.py"
click node_whisper "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/backbones/asr_whisper.py"
click node_vsr "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/backbones/vsr_avhubert.py"
click node_voice_encoder "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/voice_encoder.py"
click node_face_encoder "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/face_encoder.py"
click node_identity_pool "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/identity_pool.py"
click node_identity_gate "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/gate.py"
click node_enrollment "https://github.com/troyhow0413/avsd_ger/blob/main/one_go/c1_enroll.py"
click node_aligner "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/id_conditioned_aligner.py"
click node_prompt_builder "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/prompt_builder.py"
click node_ger_head "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/ger_head.py"
click node_soft_bridge "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/soft_token_bridge.py"
click node_confidence "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c3_feedback/confidence_scorer.py"
click node_closed_loop "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c3_feedback/closed_loop.py"
click node_safety "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c3_feedback/ger_safety.py"
click node_metrics "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/eval/metrics.py"
click node_ctc_loss "https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/training/ctc_loss.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_session_runner,node_pipeline,node_metrics toneBlue
class node_voice_encoder,node_face_encoder,node_identity_pool,node_identity_gate,node_enrollment toneAmber
class node_mouth_roi,node_frontend_registry,node_whisper,node_vsr,node_aligner,node_prompt_builder,node_ger_head,node_soft_bridge,node_ctc_loss toneMint
class node_confidence,node_closed_loop,node_safety toneRose
class node_user toneIndigo
```

## Components

### Session workflow

- [Turn runner](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/eval/session.py): `avsd_ger/eval/session.py`
- [AVSD-GER pipeline](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/pipeline.py): `avsd_ger/pipeline.py`
- [Evaluation metrics](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/eval/metrics.py): `avsd_ger/eval/metrics.py`

### C1 Identity

- [Voice encoder](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/voice_encoder.py): `avsd_ger/c1_identity/voice_encoder.py`
- [Face encoder](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/face_encoder.py): `avsd_ger/c1_identity/face_encoder.py`
- [Identity pool](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/identity_pool.py): `avsd_ger/c1_identity/identity_pool.py`
- [Identity gate](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c1_identity/gate.py): `avsd_ger/c1_identity/gate.py`
- [Speaker enrollment](https://github.com/troyhow0413/avsd_ger/blob/main/one_go/c1_enroll.py): `one_go/c1_enroll.py`

### Recognition and C2

- [Mouth ROI](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/frontend/mouth_roi.py): `avsd_ger/frontend/mouth_roi.py`
- [Frontend registry](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/frontend/registry.py): `avsd_ger/frontend/registry.py`
- [Whisper ASR](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/backbones/asr_whisper.py): `avsd_ger/backbones/asr_whisper.py`
- [AV-HuBERT VSR](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/backbones/vsr_avhubert.py): `avsd_ger/backbones/vsr_avhubert.py`
- [ID-conditioned aligner](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/id_conditioned_aligner.py): `avsd_ger/c2_alignment/id_conditioned_aligner.py`
- [Prompt builder](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/prompt_builder.py): `avsd_ger/c2_alignment/prompt_builder.py`
- [GER generator](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/ger_head.py): `avsd_ger/c2_alignment/ger_head.py`
- [Soft-token bridge](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c2_alignment/soft_token_bridge.py): `avsd_ger/c2_alignment/soft_token_bridge.py`
- [CTC training head](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/training/ctc_loss.py): `avsd_ger/training/ctc_loss.py`

### C3 Feedback

- [Confidence scorer](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c3_feedback/confidence_scorer.py): `avsd_ger/c3_feedback/confidence_scorer.py`
- [Feedback loop](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c3_feedback/closed_loop.py): `avsd_ger/c3_feedback/closed_loop.py`
- [GER safety gate](https://github.com/troyhow0413/avsd_ger/blob/main/avsd_ger/c3_feedback/ger_safety.py): `avsd_ger/c3_feedback/ger_safety.py`

### Other components

- **Session input**

## Connections

- Session input → Turn runner: provides turns
- Turn runner → AVSD-GER pipeline: processes each turn
- AVSD-GER pipeline → Mouth ROI: extracts mouth input
- AVSD-GER pipeline → Frontend registry: selects frontend
- AVSD-GER pipeline → Whisper ASR: recognizes audio
- AVSD-GER pipeline → AV-HuBERT VSR: recognizes visual speech
- AVSD-GER pipeline → Voice encoder: encodes voice
- AVSD-GER pipeline → Face encoder: encodes face
- AVSD-GER pipeline → Identity pool: queries identity
- AVSD-GER pipeline → Identity gate: gates identity
- Speaker enrollment → Voice encoder: encodes enrollment
- Speaker enrollment → Face encoder: encodes enrollment
- Speaker enrollment → Identity pool: enrolls speaker
- AVSD-GER pipeline → ID-conditioned aligner: aligns ASR features
- AVSD-GER pipeline → Prompt builder: builds correction prompt
- AVSD-GER pipeline → GER generator: generates correction
- AVSD-GER pipeline → Soft-token bridge: bridges aligned features
- AVSD-GER pipeline → Confidence scorer: scores hypotheses
- AVSD-GER pipeline → Feedback loop: checks feedback action
- AVSD-GER pipeline → GER safety gate: checks generated text
- Feedback loop → Identity pool: requests pool update
- Turn runner → Evaluation metrics: evaluates results
- CTC training head → ID-conditioned aligner: trains aligned features