# yeung-lab/harmoni architecture

> Architecture diagram of the GitHub repository yeung-lab/harmoni, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yeung-lab/harmoni
- Repository: https://github.com/yeung-lab/harmoni

## Overview

##### Project Type and Purpose
- HARMONI is a research-oriented, multi-modal processing pipeline for analyzing caregiver–child interactions in video and audio.
- It is **not** a typical client-server web app but a **batch/DAG pipeline** that fuses:
- Audio processing (speech segmentation, speaker diarization with ALICE/pyannote/VBx)
- Downstream analysis (behavior metrics) and visualization (GIF/video overlays)

##### README for Technical Stack
- Python 3.9, CUDA 11.3, NVIDIA GPU
- Conda environments: `harmoni_visual`, `process_audio`
- External dependencies: OpenPose, MiDaS, Panoptic-DeepLab, PHALP, PARE, size_depth_disambiguation
- Configuration via YAML (`data/cfgs/harmoni.yaml`, `audio/process_audio.yml`)

##### Main Components & Interactions
a. **Inputs & Config**
- Video file (e.g. GIF/MP4)
- Audio stream extracted from video
- YAML config for visual (`harmoni.yaml`) and audio pipeline (`process_audio.yml`)
- Model checkpoints (SMPL, depth, segmentation, diarization, voice-type)

b. **Visual Processing Pipeline** (invoked by `python main.py …`)
1. Preprocessing
- Video → frames → shot detection → downsample
- Depth map estimation (MiDaS)
- Ground plane estimation
2. Detection
- 2D keypoints (OpenPose)
- Semantic segmentation (Panoptic-DeepLab)
- Body‐type classification (adult vs. child)
3. Tracking
- PHALP: associate detections across frames → 3D tracklets
- DAPA / SMPL fitting → 3D joints + meshes
5. Postprocessing
- SMPLify refinement
- One-Euro filter temporal smoothing
6. Downstream Analysis
- Compute interaction metrics (e.g. distances, gaze)
7. Visualization
- Render 3D meshes/keypoints → GIF/MP4
- Overlay downstream metrics

c. **Audio Processing Pipeline** (`cd audio; python run.py`)
1. Extract audio from video
2. Feature extraction (MFCCs, fbanks)
3. Speaker diarization
- ALICE (core diarization)
- VBx and x-vector processing (VBx directory)
- pyannote-audio pipelines for overlap/speech detection
4. Voice-type classification (child vs. adult)
5. Downstream audio stats (utterance durations, turn-takings)
6. Output transcripts and RTTM files

##### Architectural Patterns & Design Principles
- **Pipeline / DAG** of discrete, reusable components
- **Modular separation**: each stage lives in its own directory with clear API
- **Configuration over code**: behavior switched via YAML flags, CLI args
- **External model / data stores**: checkpoints, config files treated as data artifacts
- **Batch processing**: entire clip processed end-to-end in one job

##### Elements
- Processing modules (blue)
- Data artifacts (green)
- External services/libs (orange)

Putting It All Together
- At the end, converge into “Downstream Analysis” → “Visualization” (visual) or “Audio Stats” (audio) → “Results Folder”
- Along the bottom, add **Infrastructure**: GPU, Conda envs, Storage, External models/GL libraries

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration
    Main["CLI: python main.py …"]:::process
    AudioOrch["CLI: python audio/run.py"]:::process

    click Main "https://github.com/yeung-lab/harmoni/blob/main/main.py"
    click AudioOrch "https://github.com/yeung-lab/harmoni/blob/main/audio/run.py"

    %% Visual Pipeline Swimlane
    Main --> Pre["Preprocessing"]:::process
    Pre --> ImgUtils["Image Utils"]:::process
    Pre --> VidUtils["Video Utils"]:::process
    Pre --> GroundNormal["Ground Plane Estimation"]:::process

    click ImgUtils "https://github.com/yeung-lab/harmoni/blob/main/utils/img_utils.py"
    click VidUtils "https://github.com/yeung-lab/harmoni/blob/main/utils/vid_utils.py"
    click GroundNormal "https://github.com/yeung-lab/harmoni/blob/main/detectors/ground_normal/get_ground_normal.py"
    click GroundNormal "https://github.com/yeung-lab/harmoni/blob/main/detectors/ground_normal/utils.py"

    Pre --> Det["Detection"]:::process
    Det --> OpenPose["OpenPose"]:::external
    Det --> MiDaS["MiDaS Depth Estimation"]:::external
    Det --> Panoptic["Panoptic-DeepLab"]:::external
    Det --> BodyType["Body-Type Classifier"]:::process
    Det --> GroundingDINO["GroundingDINO"]:::external

    click OpenPose "https://github.com/yeung-lab/harmoni/tree/main/detectors/openpose/"
    click MiDaS "https://github.com/yeung-lab/harmoni/tree/main/detectors/midas/"
    click Panoptic "https://github.com/yeung-lab/harmoni/tree/main/detectors/panoptic_deeplab/"
    click BodyType "https://github.com/yeung-lab/harmoni/blob/main/detectors/classifier/body_type_classifier.py"
    click GroundingDINO "https://github.com/yeung-lab/harmoni/tree/main/detectors/grounded_dino/"

    Det --> Track["Tracking (PHALP)"]:::external
    click Track "https://github.com/yeung-lab/harmoni/tree/main/trackers/phalp/"

    Track --> HPS["Human Pose & Shape"]:::process
    HPS --> DAPA["DAPA Fitting"]:::process
    HPS --> SMPL["SMPL Fitting"]:::process
    HPS --> Cliff["CLiFF Backbones"]:::external

    click DAPA "https://github.com/yeung-lab/harmoni/blob/main/hps/dapa/dapa.py"
    click SMPL "https://github.com/yeung-lab/harmoni/blob/main/hps/smpla.py"
    click SMPL "https://github.com/yeung-lab/harmoni/blob/main/hps/body_model.py"
    click SMPL "https://github.com/yeung-lab/harmoni/blob/main/hps/smpl_utils.py"
    click Cliff "https://github.com/yeung-lab/harmoni/tree/main/hps/cliff/"

    HPS --> Post["Postprocessing"]:::process
    Post --> SMPLify["Temporal SMPLify"]:::process
    Post --> OneEuro["One-Euro Filter"]:::process
    Post --> FilterFrames["Frame Filtering"]:::process
    Post --> PostConsts["Postprocess Constants"]:::data
    Post --> Losses["Postprocess Losses"]:::data
    Post --> Prior["Postprocess Priors"]:::data

    click SMPLify "https://github.com/yeung-lab/harmoni/blob/main/postprocess/temporal_smplify.py"
    click OneEuro "https://github.com/yeung-lab/harmoni/blob/main/postprocess/one_euro_filter.py"
    click FilterFrames "https://github.com/yeung-lab/harmoni/blob/main/postprocess/filter_frames.py"
    click PostConsts "https://github.com/yeung-lab/harmoni/blob/main/postprocess/constants.py"
    click Losses "https://github.com/yeung-lab/harmoni/blob/main/postprocess/losses.py"
    click Prior "https://github.com/yeung-lab/harmoni/blob/main/postprocess/prior.py"

    Post --> DownVisMerge["Downstream Analysis"]:::process
    DownVisMerge --> CalcDown["Compute Behavior Metrics"]:::process
    click CalcDown "https://github.com/yeung-lab/harmoni/blob/main/downstream/calc_downstream.py"

    CalcDown --> Viz["Visualization"]:::process
    Viz --> PyRender["PyRenderer"]:::external
    Viz --> Keypoints["Keypoint Drawing"]:::external
    Viz --> VizUtils["Viz Utils"]:::process
    Viz --> PHALPViz["PHALP Visualization"]:::external

    click PyRender "https://github.com/yeung-lab/harmoni/blob/main/visualization/pyrender.py"
    click Keypoints "https://github.com/yeung-lab/harmoni/blob/main/visualization/keypoints.py"
    click VizUtils "https://github.com/yeung-lab/harmoni/blob/main/visualization/utils.py"
    click PHALPViz "https://github.com/yeung-lab/harmoni/blob/main/trackers/phalp/visualize/py_renderer.py"
    click PHALPViz "https://github.com/yeung-lab/harmoni/blob/main/trackers/phalp/visualize/visualizer.py"

    %% Audio Pipeline Swimlane
    AudioOrch --> APre["Audio Preprocessing"]:::process
    APre --> AudioProcess["Process Audio"]:::process
    click AudioProcess "https://github.com/yeung-lab/harmoni/blob/main/audio/process_audio.py"
    APre --> AudioUtils["Audio Utils"]:::process
    click AudioUtils "https://github.com/yeung-lab/harmoni/blob/main/audio/utils.py"
    APre --> AudioYML["Audio Config"]:::data
    click AudioYML "https://github.com/yeung-lab/harmoni/blob/main/audio/process_audio.yml"

    APre --> Feature["Feature Extraction (MFCC)"]:::process
    Feature --> Diar["Speaker Diarization"]:::external
    Diar --> ALICE["ALICE Pipeline"]:::external
    Diar --> VBx["VBx Pipeline"]:::external
    click ALICE "https://github.com/yeung-lab/harmoni/tree/main/audio/ALICE/"
    click VBx "https://github.com/yeung-lab/harmoni/tree/main/audio/VBx/"

    Diar --> VoiceType["Voice-Type Classifier"]:::process
    VoiceType --> AudioStats["Audio Stats"]:::process
    AudioStats --> Transcripts["Transcripts & RTTM"]:::data

    %% Evaluation & Config
    Main --> Eval["Evaluation Utilities"]:::external
    click Eval "https://github.com/yeung-lab/harmoni/tree/main/evaluation/cmu_panoptic/"

    Main --> Configs["Global Config & Constants"]:::data
    click Configs "https://github.com/yeung-lab/harmoni/blob/main/constants.py"
    click Configs "https://github.com/yeung-lab/harmoni/blob/main/cmd_parser.py"
    click Configs "https://github.com/yeung-lab/harmoni/blob/main/dataset.py"

    Main --> Demo["Demo & Setup Scripts"]:::process
    click Demo "https://github.com/yeung-lab/harmoni/blob/main/install_visual.sh"
    click Demo "https://github.com/yeung-lab/harmoni/blob/main/run_demo.sh"
    click Demo "https://github.com/yeung-lab/harmoni/blob/main/results.py"

    %% Infrastructure
    classDef process fill:#a8d0e6,stroke:#333,stroke-width:1px
    classDef data fill:#b8e6a3,stroke:#333,stroke-width:1px
    classDef external fill:#f7c57c,stroke:#333,stroke-width:1px
    classDef infra fill:#cccccc,stroke:#333,stroke-width:1px

    Storage["File Storage"]:::infra
    GPU["NVIDIA GPU (CUDA 11.3)"]:::infra
    Conda["Conda Envs"]:::infra

    Storage --- Main
    GPU --- Main
    Conda --- Main
```
