# socaity/py_audio2face architecture

> Architecture diagram of the GitHub repository socaity/py_audio2face, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/socaity/py_audio2face
- Repository: https://github.com/socaity/py_audio2face

## Overview

##### Project type and purpose
- py_audio2face is an open-source Python client library/tool that drives NVIDIA’s Audio2Face headless server to generate facial animations from audio. It supports both file-based workflows (via a REST API) and real-time streaming (via gRPC).

##### File structure
- Top-level:
- py_audio2face/: the main package
- tests/: unit tests for core functionality
- pyproject.toml, MANIFEST.in: packaging/config
- Inside py_audio2face/:
- audio2face.py: entry point, Audio2Face class with high-level methods
- settings.py, utils.py: global settings and helpers
- modules/: encapsulates functional sub-systems
– _general.py: generic server control (start, shutdown)
– _audio2emotion.py: emotion detection logic
– _export.py: post-processing and USD export
– _player.py: playback controls (REST)
– _streaming.py: streaming via gRPC
– clients/: HTTP & gRPC stubs (requests-based HTTP client + autogenerated protobuf stubs)
- assets/: bundled USD scene templates (e.g., mark_arkit_solved_default.usd)

##### README for additional insights
- Describes both “single‐file” and “folder” batch processing via a2f.audio2face_single/folder
- Describes emotion configuration and streaming usage via media-toolkit and gRPC
- Highlights use cases: game-character, virtual avatars, USD export, real-time animation

a. Main components
- Client Application (your Python script importing py_audio2face)
- py_audio2face Library
– High-level API (Audio2Face class)
– Core modules (_general, _audio2emotion, _export, _player, _streaming)
– HTTP Client module
– gRPC Client stubs
– Local assets (USD scenes)
- Audio2Face Headless Server
– REST API endpoint
– gRPC endpoint for streaming
- File System
– Input: audio files or audio data stream
– Output: USD animation files
- Optional: Media-Toolkit (audio generator) or other audio sources
- Downstream consumers (Maya/Unreal Engine)

b. Relationships and interactions
1. User script → calls py_audio2face.Audio2Face methods
2. Audio2Face.init() → loads a USD scene (assets) on the remote server
3. For file mode: HTTP POST audio file path → server processes → REST GET to fetch animation → _export writes USD
4. For streaming mode: establish gRPC channel → stream audio frames via gRPC client stub → server streams blendshape frames back → _export writes USD or pushes frames onward
5. Emotion sub-module may analyze audio via local logic or remote service call before or during streaming

c. Architectural patterns & design principles
- Modular layering: separation of concerns between control (general), emotion, export, playback, streaming
- Adapter pattern: HTTP client and gRPC stub modules abstract transport details
- Template reuse: USD scene templates in assets folder
- Configurable settings: centralized in settings.py

d. Key technologies/frameworks
- Python 3.x
- requests (HTTP client)
- grpcio, protobuf (gRPC streaming)
- USD (Universal Scene Description) for animation export
- NVIDIA Audio2Face REST & gRPC APIs

##### Putting it all together
1. Top: “User Application” calls into
2. Middle: “py_audio2face Library” subdivided into five modules + clients + assets
3. Right: “Audio2Face Headless Server” with two APIs (REST, gRPC)
4. Bottom: “File System” (audio in, USD out)
- Key flows—single file mode vs streaming mode—and note where emotion detection, export, and playback modules intervene.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Top-level user application
    UserApp["User Application"]:::external

    %% py_audio2face Library
    subgraph "py_audio2face Library"
        direction TB
        Audio2Face["Audio2Face Class"]:::library
        Settings["Global Settings"]:::library
        Utils["Utility Helpers"]:::library

        subgraph "Modules"
            direction TB
            General["Generic Server Control\n(_general)"]:::library
            Emotion["Emotion Detection\n(_audio2emotion)"]:::library
            Export["Export / USD Writing\n(_export)"]:::library
            Player["REST Playback Control\n(_player)"]:::library
            StreamingMod["Streaming Module\n(_streaming)"]:::library
        end

        subgraph "Clients"
            direction TB
            HTTPClient["HTTP Client Adapter"]:::client
            GrpcMsgs["gRPC Stub Messages\n(audio2face_pb2)"]:::client
            GrpcSvc["gRPC Stub Services\n(audio2face_pb2_grpc)"]:::client
        end

        subgraph "Assets"
            direction TB
            USDDefault["mark_arkit_solved_default.usd"]:::asset
            USDStream["mark_arkit_solved_streaming.usd"]:::asset
        end
    end

    %% External server
    subgraph "Audio2Face Headless Server"
        direction TB
        RESTAPI["REST API Endpoint"]:::server
        GRPCAPI["gRPC Endpoint"]:::server
    end

    %% File system
    subgraph "File System"
        direction TB
        AudioIn["Audio Input (files/stream)"]:::fileIO
        USDOut["USD Animation Output"]:::fileIO
    end

    %% Flows
    UserApp -->|"calls"| Audio2Face
    Audio2Face --> Settings
    Audio2Face --> Utils

    %% File-based workflow
    Audio2Face -->|"file mode"| HTTPClient
    HTTPClient -->|"POST audio"| RESTAPI
    RESTAPI -->|"animation data"| Player
    Player --> Export
    Export --> USDOut

    %% Streaming workflow
    Audio2Face -->|"stream mode"| StreamingMod
    StreamingMod --> GrpcMsgs
    GrpcMsgs --> GrpcSvc
    GrpcSvc -->|"stream audio"| GRPCAPI
    GRPCAPI -->|"stream shapes"| StreamingMod
    StreamingMod --> Export
    Export --> USDOut

    %% Emotion detection
    Audio2Face --> Emotion
    Emotion --> RESTAPI
    Emotion --> GRPCAPI

    %% Assets loading
    Audio2Face --> USDDefault
    StreamingMod --> USDStream

    %% File system input
    AudioIn --> Audio2Face
    AudioIn --> StreamingMod

    %% Click events
    click Audio2Face "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/audio2face.py"
    click Settings "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/settings.py"
    click Utils "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/utils.py"
    click General "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/_general.py"
    click Emotion "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/_audio2emotion.py"
    click Export "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/_export.py"
    click Player "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/_player.py"
    click StreamingMod "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/_streaming.py"
    click HTTPClient "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/clients/_http_client.py"
    click GrpcMsgs "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/clients/grpc_stub/audio2face_pb2.py"
    click GrpcSvc "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/modules/clients/grpc_stub/audio2face_pb2_grpc.py"
    click USDDefault "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/assets/mark_arkit_solved_default.usd"
    click USDStream "https://github.com/socaity/py_audio2face/blob/main/py_audio2face/assets/mark_arkit_solved_streaming.usd"

    %% Styles
    classDef library fill:#bbdefb,stroke:#1e88e5,stroke-width:1px
    classDef client fill:#b3e5fc,stroke:#0288d1,stroke-width:1px
    classDef asset fill:#e1bee7,stroke:#8e24aa,stroke-width:1px
    classDef server fill:#c8e6c9,stroke:#388e3c,stroke-width:1px
    classDef fileIO fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
    classDef external fill:#eeeeee,stroke:#757575,stroke-width:1px
```
