# emilianavt/openseeface architecture

> Architecture diagram of the GitHub repository emilianavt/openseeface, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/emilianavt/openseeface
- Repository: https://github.com/emilianavt/openseeface

## Overview

##### Project type and purpose
- This is a cross-platform face-tracking library and toolkit, not a standalone application.
- Core purpose: detect and track facial landmarks in video (webcam or file), then ship tracking data over UDP for real-time avatar puppeteering in Unity (or any other client).
- Secondary features: expression classification (SVM-based), process management (launching the tracker from Unity), example integrations, and C++ wrappers for custom SVM backends (ThunderSVM).

##### File structure
- Top-level:
- .github/ → CI workflows to build Python binaries (pyinstaller), run tests, upload artifacts
- License files and third-party LICENSES folder
- README.md, pyproject.toml, build scripts (make_exe.{sh,bat})
- Python scripts: facetracker.py (main), retinaface.py, tracker.py, model.py, remedian.py, dshowcapture.py, escapi.py, input_reader.py, similaritytransform.py
- models/ → ONNX landmark and detection models, priorbox JSON
- Source/ → C++ wrappers for libsvm and ThunderSVM
- Unity/ → C# plugins and components: OpenSeeLauncher, OpenSee, OpenSeeExpression, IK targets, visualization (OpenSeeShowPoints)
- Examples/ → Unity C# example scripts and sample animations
- Key patterns:
- Modular Python: separate scripts for detection, tracking, IPC, model definitions
- IPC via UDP socket
- Process-management in Unity using WinAPI job objects
- C++ “SVMWrapper” modules compiled into Python extension modules or Unity plugins
- ONNX Runtime for inference

##### Readme insights
- System comprised of a face-tracking “server” (facetracker.py/EXE) and “client” (Unity plugin) communicating over UDP.
- CLI options for model selection, threading, input type (webcam vs. video), visualization.
- Unity side: receive packets in a background thread, expose them via public fields to C# components.
- Expression detection uses LIBSVM via C++ wrappers.
- Build pipeline: Poetry/pyproject.toml for Python deps, pyinstaller to produce facetracker.exe, .spec file for packaging, GitHub Actions to automate.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Python Tracker
    subgraph "Python Tracker"
        subgraph "Video Input Module"
            OpenCV["OpenCV Capture"]:::python
            DShow["dshowcapture"]:::python
            ESCAPI["escapi"]:::python
            InputReader["input_reader.py"]:::python
            FacetrackerEntry["facetracker.py"]:::python
            click DShow "https://github.com/emilianavt/openseeface/tree/master/dshowcapture/"
            click ESCAPI "https://github.com/emilianavt/openseeface/tree/master/escapi/"
            click InputReader "https://github.com/emilianavt/openseeface/blob/master/input_reader.py"
            click FacetrackerEntry "https://github.com/emilianavt/openseeface/blob/master/facetracker.py"
        end
        subgraph "Face Detection Module"
            Retina["RetinaFace ONNX"]:::models
            PriorBox["PriorBox JSON"]:::models
            faceDetectScript["retinaface.py"]:::python
            click faceDetectScript "https://github.com/emilianavt/openseeface/blob/master/retinaface.py"
            click Retina "https://github.com/emilianavt/openseeface/blob/master/models/retinaface_640x640_opt.onnx"
            click PriorBox "https://github.com/emilianavt/openseeface/blob/master/models/priorbox_640x640.json"
        end
        subgraph "Landmark Tracking Module"
            TrackerScript["tracker.py"]:::python
            ModelScript["model.py"]:::python
            Similarity["similaritytransform.py"]:::python
            Remedian["remedian.py"]:::python
            MNV3Det["mnv3_detection_opt.onnx"]:::models
            LMModels["lm_model*_opt.onnx"]:::models
            Gaze["mnv3_gaze32_split_opt.onnx"]:::models
            click TrackerScript "https://github.com/emilianavt/openseeface/blob/master/tracker.py"
            click ModelScript "https://github.com/emilianavt/openseeface/blob/master/model.py"
            click Similarity "https://github.com/emilianavt/openseeface/blob/master/similaritytransform.py"
            click Remedian "https://github.com/emilianavt/openseeface/blob/master/remedian.py"
            click MNV3Det "https://github.com/emilianavt/openseeface/blob/master/models/mnv3_detection_opt.onnx"
            click Gaze "https://github.com/emilianavt/openseeface/blob/master/models/mnv3_gaze32_split_opt.onnx"
        end
        subgraph "Expression Classifier"
            SVMWrapperCpp["SVMWrapper.cpp"]:::python
            ThunderCpp["ThunderSVMWrapper.cpp"]:::python
            svm_c["svm.cpp"]:::python
            svm_h["svm.h"]:::python
            click SVMWrapperCpp "https://github.com/emilianavt/openseeface/blob/master/Source/SVMWrapper.cpp"
            click ThunderCpp "https://github.com/emilianavt/openseeface/blob/master/Source/ThunderSVMWrapper.cpp"
            click svm_c "https://github.com/emilianavt/openseeface/blob/master/Source/svm.cpp"
            click svm_h "https://github.com/emilianavt/openseeface/blob/master/Source/svm.h"
        end
    end

    subgraph "IPC Layer"
        UDPTransmitter["UDP Transmitter"]:::ipc
        click UDPTransmitter "https://github.com/emilianavt/openseeface/blob/master/facetracker.py"
    end

    subgraph "Unity Client"
        Launcher["OpenSeeLauncher"]:::unity
        JobMgmt["OpenSeeJobManagement"]:::unity
        OpenSeeComp["OpenSee.cs"]:::unity
        ProcessIF["OpenSeeProcessInterface.cs"]:::unity
        ShowPoints["OpenSeeShowPoints.cs"]:::unity
        ExpressionComp["OpenSeeExpression.cs"]:::unity
        IKTarget["OpenSeeIKTarget.cs"]:::unity
        Interpolation["OpenSeeInterpolation.cs"]:::unity
        KinInterp["OpenSeeKinematicInterpolation.cs"]:::unity
        SVMModel["SVMModel.cs"]:::unity
        ThunderModel["ThunderSVMModel.cs"]:::unity
        click Launcher "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeLauncher.cs"
        click JobMgmt "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeJobManagement.cs"
        click OpenSeeComp "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSee.cs"
        click ProcessIF "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeProcessInterface.cs"
        click ShowPoints "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeShowPoints.cs"
        click ExpressionComp "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeExpression.cs"
        click IKTarget "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeIKTarget.cs"
        click Interpolation "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeInterpolation.cs"
        click KinInterp "https://github.com/emilianavt/openseeface/blob/master/Unity/OpenSeeKinematicInterpolation.cs"
        click SVMModel "https://github.com/emilianavt/openseeface/blob/master/Unity/Plugins/SVMModel.cs"
        click ThunderModel "https://github.com/emilianavt/openseeface/blob/master/Unity/Plugins/ThunderSVMModel.cs"
    end

    subgraph "Build & CI"
        GHActions["GitHub Actions"]:::build
        MakeSh["make_exe.sh"]:::build
        MakeBat["make_exe.bat"]:::build
        Spec["facetracker.spec"]:::build
        PyProj["pyproject.toml"]:::build
        click GHActions "https://github.com/emilianavt/openseeface/blob/master/.github/workflows/create-binaries.yaml"
        click MakeSh "https://github.com/emilianavt/openseeface/blob/master/make_exe.sh"
        click MakeBat "https://github.com/emilianavt/openseeface/blob/master/make_exe.bat"
        click Spec "https://github.com/emilianavt/openseeface/blob/master/facetracker.spec"
        click PyProj "https://github.com/emilianavt/openseeface/blob/master/pyproject.toml"
    end

    %% Data Flows
    OpenCV -->|camera feed| Retina
    DShow -->|camera feed| Retina
    ESCAPI -->|camera feed| Retina
    InputReader -->|file input| Retina
    Retina -->|"face bboxes"| TrackerScript
    TrackerScript -->|"landmarks"| ModelScript
    ModelScript -->|"transform"| Remedian
    Remedian -->|"filtered landmarks"| Similarity
    Similarity -->|"landmarks"| SVMWrapperCpp
    Similarity -->|"landmarks"| ThunderCpp
    SVMWrapperCpp -->|"expression labels"| UDPTransmitter
    ThunderCpp -->|"expression labels"| UDPTransmitter
    TrackerScript -->|"trackingData JSON/binary"| UDPTransmitter

    UDPTransmitter -->|"UDP: trackingData"| OpenSeeComp
    OpenSeeComp -->|"parsed data"| ExpressionComp
    OpenSeeComp -->|"parsed data"| IKTarget
    ExpressionComp -->|"blend weights"| ShowPoints
    IKTarget -->|"target rotation"| ShowPoints
    JobMgmt -->|manages| Launcher
    Launcher -->|spawns| FacetrackerEntry

    GHActions -->|build exe| FacetrackerEntry
    MakeSh -->|packaging| FacetrackerEntry
    MakeBat -->|packaging| FacetrackerEntry
    Spec -->|pyinstaller spec| FacetrackerEntry
    PyProj -->|deps| FacetrackerEntry

    %% Styles
    classDef python fill:#add8e6,stroke:#000,stroke-width:1px
    classDef models fill:#90ee90,stroke:#000,stroke-width:1px
    classDef ipc fill:#ffffe0,stroke:#000,stroke-width:1px
    classDef unity fill:#dda0dd,stroke:#000,stroke-width:1px
    classDef build fill:#d3d3d3,stroke:#000,stroke-width:1px
```
