How emilianavt/openseeface works
Project type and purpose
• This is a cross-platform face-tracking library and toolkit, not a standalone application.
• Core purpose: detect and track facial landmarks in video (webcam or file), then ship tracking data over UDP for real-time avatar puppeteering in Unity (or any other client).
• Secondary features: expression classification (SVM-based), process management (launching the tracker from Unity), example integrations, and C++ wrappers for custom SVM backends (ThunderSVM).
File structure
• Top-level:
• .github/ → CI workflows to build Python binaries (pyinstaller), run tests, upload artifacts
• License files and third-party LICENSES folder
• README.md, pyproject.toml, build scripts (make_exe.{sh,bat})
• Python scripts: facetracker.py (main), retinaface.py, tracker.py, model.py, remedian.py, dshowcapture.py, escapi.py, input_reader.py, similaritytransform.py
• models/ → ONNX landmark and detection models, priorbox JSON
• Source/ → C++ wrappers for libsvm and ThunderSVM
• Unity/ → C# plugins and components: OpenSeeLauncher, OpenSee, OpenSeeExpression, IK targets, visualization (OpenSeeShowPoints)
• Examples/ → Unity C# example scripts and sample animations
• Key patterns:
• Modular Python: separate scripts for detection, tracking, IPC, model definitions
• IPC via UDP socket
• Process-management in Unity using WinAPI job objects
• C++ “SVMWrapper” modules compiled into Python extension modules or Unity plugins
• ONNX Runtime for inference
Readme insights
• System comprised of a face-tracking “server” (facetracker.py/EXE) and “client” (Unity plugin) communicating over UDP.
• CLI options for model selection, threading, input type (webcam vs. video), visualization.
• Unity side: receive packets in a background thread, expose them via public fields to C# components.
• Expression detection uses LIBSVM via C++ wrappers.
• Build pipeline: Poetry/pyproject.toml for Python deps, pyinstaller to produce facetracker.exe, .spec file for packaging, GitHub Actions to automate.