How troyhow0413/avsd_ger works
AVSD-GER performs identity-conditioned, closed-loop correction of audio-visual speech recognition for multi-speaker sessions. Its principal workflow processes ordered session turns through a per-turn pipeline: prepare visual mouth input when needed, obtain audio and visual hypotheses, retrieve an enrolled speaker identity, align ASR features using identity, generate a corrected transcript, and apply confidence-gated feedback that may update the identity pool. The graph also preserves the documented enrollment and evaluation roles. Pipeline and session excerpts are partial, so links between stages follow their explicit module descriptions and documented design; uncertain internal call details are omitted.
Main entry points: pipeline.py AVSD-GER pipeline · c1_enroll.py Speaker enrollment · session.py Turn runner · identity_pool.py Identity pool
Components 21
Session workflow
- Turn runner
avsd_ger/eval/session.py - AVSD-GER pipeline
avsd_ger/pipeline.py - Evaluation metrics
avsd_ger/eval/metrics.py
C1 Identity
- Voice encoder
avsd_ger/c1_identity/voice_encoder.py - Face encoder
avsd_ger/c1_identity/face_encoder.py - Identity pool
avsd_ger/c1_identity/identity_pool.py - Identity gate
avsd_ger/c1_identity/gate.py - Speaker enrollment
one_go/c1_enroll.py
Recognition and C2
- Mouth ROI
avsd_ger/frontend/mouth_roi.py - Frontend registry
avsd_ger/frontend/registry.py - Whisper ASR
avsd_ger/backbones/asr_whisper.py - AV-HuBERT VSR
avsd_ger/backbones/vsr_avhubert.py - ID-conditioned aligner
avsd_ger/c2_alignment/id_conditioned_aligner.py - Prompt builder
avsd_ger/c2_alignment/prompt_builder.py - GER generator
avsd_ger/c2_alignment/ger_head.py - Soft-token bridge
avsd_ger/c2_alignment/soft_token_bridge.py - CTC training head
avsd_ger/training/ctc_loss.py
C3 Feedback
- Confidence scorer
avsd_ger/c3_feedback/confidence_scorer.py - Feedback loop
avsd_ger/c3_feedback/closed_loop.py - GER safety gate
avsd_ger/c3_feedback/ger_safety.py
Other components
- Session input
Connections 23
- Session input → Turn runner: provides turns
- Turn runner → AVSD-GER pipeline: processes each turn
- AVSD-GER pipeline → Mouth ROI: extracts mouth input
- AVSD-GER pipeline → Frontend registry: selects frontend
- AVSD-GER pipeline → Whisper ASR: recognizes audio
- AVSD-GER pipeline → AV-HuBERT VSR: recognizes visual speech
- AVSD-GER pipeline → Voice encoder: encodes voice
- AVSD-GER pipeline → Face encoder: encodes face
- AVSD-GER pipeline → Identity pool: queries identity
- AVSD-GER pipeline → Identity gate: gates identity
- Speaker enrollment → Voice encoder: encodes enrollment
- Speaker enrollment → Face encoder: encodes enrollment
- Speaker enrollment → Identity pool: enrolls speaker
- AVSD-GER pipeline → ID-conditioned aligner: aligns ASR features
- AVSD-GER pipeline → Prompt builder: builds correction prompt
- AVSD-GER pipeline → GER generator: generates correction
- AVSD-GER pipeline → Soft-token bridge: bridges aligned features
- AVSD-GER pipeline → Confidence scorer: scores hypotheses
- AVSD-GER pipeline → Feedback loop: checks feedback action
- AVSD-GER pipeline → GER safety gate: checks generated text
- Feedback loop → Identity pool: requests pool update
- Turn runner → Evaluation metrics: evaluates results
- CTC training head → ID-conditioned aligner: trains aligned features