How yeung-lab/harmoni works
Project Type and Purpose
• HARMONI is a research-oriented, multi-modal processing pipeline for analyzing caregiver–child interactions in video and audio.
• It is not a typical client-server web app but a batch/DAG pipeline that fuses:
• Audio processing (speech segmentation, speaker diarization with ALICE/pyannote/VBx)
• Downstream analysis (behavior metrics) and visualization (GIF/video overlays)
README for Technical Stack
• Python 3.9, CUDA 11.3, NVIDIA GPU
• Conda environments: harmoni_visual, process_audio
• External dependencies: OpenPose, MiDaS, Panoptic-DeepLab, PHALP, PARE, size_depth_disambiguation
• Configuration via YAML (data/cfgs/harmoni.yaml, audio/process_audio.yml)
Main Components & Interactions
a. Inputs & Config
• Video file (e.g. GIF/MP4)
• Audio stream extracted from video
• YAML config for visual (harmoni.yaml) and audio pipeline (process_audio.yml)
• Model checkpoints (SMPL, depth, segmentation, diarization, voice-type)
b. Visual Processing Pipeline (invoked by python main.py …)
1. Preprocessing
• Video → frames → shot detection → downsample
• Depth map estimation (MiDaS)
• Ground plane estimation
2. Detection
• 2D keypoints (OpenPose)
• Semantic segmentation (Panoptic-DeepLab)
• Body‐type classification (adult vs. child)
3. Tracking
• PHALP: associate detections across frames → 3D tracklets
• DAPA / SMPL fitting → 3D joints + meshes
5. Postprocessing
• SMPLify refinement
• One-Euro filter temporal smoothing
6. Downstream Analysis
• Compute interaction metrics (e.g. distances, gaze)
7. Visualization
• Render 3D meshes/keypoints → GIF/MP4
• Overlay downstream metrics
c. Audio Processing Pipeline (cd audio; python run.py)
1. Extract audio from video
2. Feature extraction (MFCCs, fbanks)
3. Speaker diarization
• ALICE (core diarization)
• VBx and x-vector processing (VBx directory)
• pyannote-audio pipelines for overlap/speech detection
4. Voice-type classification (child vs. adult)
5. Downstream audio stats (utterance durations, turn-takings)
6. Output transcripts and RTTM files
Architectural Patterns & Design Principles
• Pipeline / DAG of discrete, reusable components
• Modular separation: each stage lives in its own directory with clear API
• Configuration over code: behavior switched via YAML flags, CLI args
• External model / data stores: checkpoints, config files treated as data artifacts
• Batch processing: entire clip processed end-to-end in one job
Elements
• Processing modules (blue)
• Data artifacts (green)
• External services/libs (orange)
Putting It All Together
• At the end, converge into “Downstream Analysis” → “Visualization” (visual) or “Audio Stats” (audio) → “Results Folder”
• Along the bottom, add Infrastructure: GPU, Conda envs, Storage, External models/GL libraries