How sb-ai-lab/emotiefflib works
Project type and purpose
• EmotiEffLib is an open-source, cross-language (Python & C++) library for real-time emotion and engagement recognition in images and videos.
• It provides a common API surface in Python and C++, pluggable backends (PyTorch & ONNX), pre-trained model artifacts, tutorials/notebooks, and CI/CD/documentation tooling.
File structure
• Top-level:
• .github, .binder, CI & environment configs
• docs/ – Doxygen & tutorial notebooks
• emotiefflib/ – Python package
• emotieffcpplib/ – C++ library with CMake, 3rdparty (OpenCV-MTCNN, xtensor, xtl)
• models/ – prebuilt weights (PyTorch .pt, Keras .h5, ONNX)
• tests/ – unit tests, shell helpers
• training_and_examples/ – Jupyter notebooks, mobile demo apps, model-prep scripts
• Patterns:
• Clear separation between core API layers and backend implementations
• Dedicated model-conversion & preparation scripts
• Multi-language coverage with shared model artifacts
• Standard Python packaging (setup.py, pyproject.toml) and C++ build via CMake
README for additional insights
• Two parallel implementations (Python & C++) share architectures and models
• Emphasis on backend abstraction: switching between ONNX runtime and native PyTorch
• Tutorial workflows (Binder & Colab) demonstrate end-to-end pipelines from image capture → face detection → feature extraction → emotion/engagement inference
• Model conversion tool (models/prepare_models_for_emotieffcpplib.py) bridges Python-trained PyTorch models to C++ compatible formats
a. Main components
• Client Interfaces
• Python API (emotiefflib/)
• C++ API (emotieffcpplib/)
• Backend Engines
• PyTorch backend
• ONNX runtime backend
• Face Detection
• OpenCV-MTCNN (3rdparty)
• Model Artifacts Store (models/)
• Conversion Tools
• Python scripts preparing models for C++
• CI/CD & Documentation
• GitHub Actions pipelines
• Binder & Conda environments
• Doxygen docs
• Tutorials & Examples
• Jupyter notebooks
• Mobile demos (Android apps + optional server component)
b. Relationships & interactions
• User code (Python or C++) → calls core API → selects backend → loads model from models/ → runs face detection → feature extraction → classification heads → outputs emotion/engagement metrics
• Model-prep script: Python PyTorch → ONNX → C++ library
• CI pipelines: triggers on PRs to build/test Python package, compile C++ library, run notebooks smoke tests
• Tutorials & Binder: launch isolated environment to showcase pipelines
c. Architectural patterns & design principles
• Backend abstraction (strategy pattern) for engine selection
• Shared data definitions (e.g., facial_analysis interfaces) across languages
• Modular design separating detection, feature extraction, and inference heads
• Language-agnostic model format (ONNX) as interoperability layer
d. Key technologies & frameworks
• Python: PyTorch, ONNX, timm, NumPy
• C++: CMake, xtensor, OpenCV (MTCNN), ONNXRuntime, libtorch
• Documentation: Doxygen, Sphinx (implied), Binder
• CI: GitHub Actions
Detail
• Distinguish between Python & C++ build/test flows
• Model conversion pipeline as a separate flow