# sb-ai-lab/emotiefflib architecture

> Architecture diagram of the GitHub repository sb-ai-lab/emotiefflib, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sb-ai-lab/emotiefflib
- Repository: https://github.com/sb-ai-lab/emotiefflib

## Overview

##### Project type and purpose
- EmotiEffLib is an open-source, cross-language (Python & C++) library for real-time emotion and engagement recognition in images and videos.
- It provides a common API surface in Python and C++, pluggable backends (PyTorch & ONNX), pre-trained model artifacts, tutorials/notebooks, and CI/CD/documentation tooling.

##### File structure
- Top-level:
- `.github`, `.binder`, CI & environment configs
- `docs/` – Doxygen & tutorial notebooks
- `emotiefflib/` – Python package
- `emotieffcpplib/` – C++ library with CMake, 3rdparty (OpenCV-MTCNN, xtensor, xtl)
- `models/` – prebuilt weights (PyTorch `.pt`, Keras `.h5`, ONNX)
- `tests/` – unit tests, shell helpers
- `training_and_examples/` – Jupyter notebooks, mobile demo apps, model-prep scripts
- Patterns:
- Clear separation between core API layers and backend implementations
- Dedicated model-conversion & preparation scripts
- Multi-language coverage with shared model artifacts
- Standard Python packaging (`setup.py`, `pyproject.toml`) and C++ build via CMake

##### README for additional insights
- Two parallel implementations (Python & C++) share architectures and models
- Emphasis on backend abstraction: switching between ONNX runtime and native PyTorch
- Tutorial workflows (Binder & Colab) demonstrate end-to-end pipelines from image capture → face detection → feature extraction → emotion/engagement inference
- Model conversion tool (`models/prepare_models_for_emotieffcpplib.py`) bridges Python-trained PyTorch models to C++ compatible formats

a. Main components
- Client Interfaces
- Python API (`emotiefflib/`)
- C++ API (`emotieffcpplib/`)
- Backend Engines
- PyTorch backend
- ONNX runtime backend
- Face Detection
- OpenCV-MTCNN (3rdparty)
- Model Artifacts Store (`models/`)
- Conversion Tools
- Python scripts preparing models for C++
- CI/CD & Documentation
- GitHub Actions pipelines
- Binder & Conda environments
- Doxygen docs
- Tutorials & Examples
- Jupyter notebooks
- Mobile demos (Android apps + optional server component)

b. Relationships & interactions
- User code (Python or C++) → calls core API → selects backend → loads model from `models/` → runs face detection → feature extraction → classification heads → outputs emotion/engagement metrics
- Model-prep script: Python PyTorch → ONNX → C++ library
- CI pipelines: triggers on PRs to build/test Python package, compile C++ library, run notebooks smoke tests
- Tutorials & Binder: launch isolated environment to showcase pipelines

c. Architectural patterns & design principles
- Backend abstraction (strategy pattern) for engine selection
- Shared data definitions (e.g., facial_analysis interfaces) across languages
- Modular design separating detection, feature extraction, and inference heads
- Language-agnostic model format (ONNX) as interoperability layer

d. Key technologies & frameworks
- Python: PyTorch, ONNX, timm, NumPy
- C++: CMake, xtensor, OpenCV (MTCNN), ONNXRuntime, libtorch
- Documentation: Doxygen, Sphinx (implied), Binder
- CI: GitHub Actions

##### Detail
- Distinguish between Python & C++ build/test flows
- Model conversion pipeline as a separate flow

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients
    subgraph "Clients"
        PythonClient["Python Client\n(pip install emotiefflib)"]:::library
        CppClient["C++ Client\n(link emotieffcpplib)"]:::library
    end

    %% Core Interfaces
    subgraph "Core Interfaces"
        PythonAPI["Python API\n(emotiefflib/)"]:::library
        subgraph "C++ API"
            CppAPI_H["Facial Analysis.h"]:::library
            CppAPI_CPP["Facial Analysis.cpp"]:::library
        end
    end

    %% Backends and Detection
    subgraph "Backends (C++) & Detection"
        Torch_H["Torch Backend.h"]:::library
        Torch_CPP["Torch Backend.cpp"]:::library
        ONNX_H["ONNX Backend.h"]:::library
        ONNX_CPP["ONNX Backend.cpp"]:::library
        FaceDetect["OpenCV-MTCNN"]:::library
    end

    %% Model Store
    ModelStore["Model Artifacts\n(models/)"]:::datastore

    %% Conversion Tools
    subgraph "Model Conversion Tools"
        Converter_gen["prepare_models_for_emotieffcpplib.py"]:::tool
        Converter_eng["engagement_single_attention_converter.py"]:::tool
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        FaceDetect --> FeatureExt["Feature Extraction"]:::library
        FeatureExt --> Classifier["Classification Heads\n(emotion & engagement)"]:::library
        Classifier --> Output["Metrics\n(emotion/engagement)"]:::library
    end

    %% Tooling & CI/CD
    CMake["CMake Build\nemotieffcpplib/CMakeLists.txt"]:::tool
    PyPack["Python Packaging\n(setup.py, pyproject.toml)"]:::tool
    Tut["Tutorials & Examples\n(docs/tutorials/, training_and_examples/)"]:::tool
    Docs["Documentation\n(docs/)"]:::ci
    CI["CI/CD Pipelines\n(.github/workflows/)"]:::ci

    %% Connections: Clients to APIs
    PythonClient -->|"import"| PythonAPI
    CppClient -->|"link & call"| CppAPI_H

    %% APIs to Pipeline
    PythonAPI -->|"calls"| FaceDetect
    PythonAPI -->|"calls"| Classifier
    CppAPI_H -->|"calls"| FaceDetect
    CppAPI_H -->|"calls"| Classifier

    %% Backends implement classification
    Torch_H -->|"implements"| Classifier
    Torch_CPP -->|"implements"| Classifier
    ONNX_H -->|"implements"| Classifier
    ONNX_CPP -->|"implements"| Classifier

    %% Model Store usage
    ModelStore -->|"provides models"| Classifier

    %% Conversion Tools produce models
    Converter_gen -->|"converts PyTorch→ONNX"| ModelStore
    Converter_eng -->|"converts engagement model"| ModelStore

    %% Build & Packaging
    CMake --> CppAPI_H
    PyPack --> PythonAPI

    %% Tutorials, Docs & CI
    Tut --> PythonAPI
    Tut --> CppAPI_H
    Docs --> PythonAPI
    Docs --> CppAPI_H
    CI --> PythonAPI
    CI --> CppAPI_H
    CI --> Tut

    %% Click Events
    click PythonAPI "https://github.com/sb-ai-lab/emotiefflib/tree/main/emotiefflib/"
    click CppAPI_H "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/include/emotiefflib/facial_analysis.h"
    click CppAPI_CPP "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/src/facial_analysis.cpp"
    click Torch_H "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/include/emotiefflib/backends/torch/facial_analysis.h"
    click Torch_CPP "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/src/backends/torch/facial_analysis.cpp"
    click ONNX_H "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/include/emotiefflib/backends/onnx/facial_analysis.h"
    click ONNX_CPP "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/src/backends/onnx/facial_analysis.cpp"
    click FaceDetect "https://github.com/sb-ai-lab/emotiefflib/tree/main/emotieffcpplib/3rdparty/opencv-mtcnn/"
    click ModelStore "https://github.com/sb-ai-lab/emotiefflib/tree/main/models/"
    click Converter_gen "https://github.com/sb-ai-lab/emotiefflib/blob/main/models/prepare_models_for_emotieffcpplib.py"
    click Converter_eng "https://github.com/sb-ai-lab/emotiefflib/blob/main/models/engagement_classification/engagement_single_attention_converter.py"
    click CMake "https://github.com/sb-ai-lab/emotiefflib/blob/main/emotieffcpplib/CMakeLists.txt"
    click PyPack "https://github.com/sb-ai-lab/emotiefflib/blob/main/setup.py"
    click Tut "https://github.com/sb-ai-lab/emotiefflib/tree/main/docs/tutorials/python/"
    click Docs "https://github.com/sb-ai-lab/emotiefflib/tree/main/docs/"
    click CI "https://github.com/sb-ai-lab/emotiefflib/tree/main/.github/workflows/"

    %% Styles
    classDef library fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef datastore fill:#90EE90,stroke:#000,stroke-width:1px
    classDef tool fill:#FFA500,stroke:#000,stroke-width:1px
    classDef ci fill:#D3D3D3,stroke:#000,stroke-width:1px
```
