# oriartyz/signactioninterpreter architecture

> Architecture diagram of the GitHub repository oriartyz/signactioninterpreter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oriartyz/signactioninterpreter
- Repository: https://github.com/oriartyz/signactioninterpreter

## Overview

##### Main Components
- Input / Data Acquisition Module: This includes the real-time video feed (from a camera or video file) and the MediaPipe Holistic tool for extracting keypoints.
- Data Preprocessing and Storage: Once keypoints are extracted, the data (saved as .npy files in various subdirectories inside MP_Data labeled “hello”, “iloveyou”, “thanks”, etc.) is preprocessed. The subfolder structure indicates labeled datasets for different sign actions.
- Logging and Monitoring: The Logs directory shows TensorBoard event files, which indicates that training progress, metrics, and visualizations are being logged.
- Model Training Module: This is represented in notebooks such as Action Detection.ipynb. Here, the pipeline includes loading the preprocessed keypoints, building the LSTM-based model (as specified in the README), training it, evaluating performance, and finally saving the trained model (action.h5).
- Model Inference / Real-time Prediction Module: Once the model is trained and saved, another component (also likely implemented in notebooks and scripts) takes live video, extracts keypoints with MediaPipe, processes them in the same way as training data, and feeds them to the trained LSTM model to predict the corresponding sign language action.
- Auxiliary Resources: The “ressource” directory (which includes an image used in the README) can be shown as supplementary material that aids in documentation and UI design for the project.

##### Relationships and Interactions
- Data flows from the Input Module to the Preprocessing Module where keypoints are extracted. Preprocessed data is then stored (shown as MP_Data) and used to train the model.
- The training process generates logs (in the Logs directory) and ultimately produces the saved model file (action.h5).
- During real-time predictions, the live data is processed through the same extraction and preprocessing pipeline and then sent to the model for interpretation.
- There is a feedback loop where the performance (e.g., evaluation runs) can trigger further retraining or fine-tuning of the model.

##### Important Architectural Patterns and Design Principles
- Modularity: Each part (data extraction, preprocessing, model training, and inference) is clearly separated, which helps maintain and upgrade the system.
- Pipeline Design: Data flows through distinct stages. This pattern resembles a typical machine learning pipeline with offline training and online prediction paths.
- Reproducibility and Logging: The use of TensorBoard logs emphasizes the importance of monitoring and reproducibility.
- Use of Notebooks: The code is organized as notebooks (with checkpoints) that make prototyping and visualization integral to development.

##### Relevant Technologies
- TensorFlow and Keras: For building, training, and saving the LSTM model.
- MediaPipe: For keypoint extraction from video.
- OpenCV: For handling video capture and processing.
- Matplotlib: For plotting and visualizing results during training or evaluation.
- Python Notebooks: Serve as a development and experimentation environment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Offline Training Pipeline"
        A1["Video Input (Camera/Video File)"]:::input
        A2["MediaPipe Holistic Extraction"]:::process
        A3["Data Preprocessing & Storage (MP_Data)"]:::data
        A4["LSTM Model Training (Action Detection.ipynb)"]:::ml
        A5["TensorBoard Logging (Logs)"]:::log
        A6["Saved Model (action.h5)"]:::model
        A1 --> A2
        A2 --> A3
        A3 --> A4
        A4 --> A5
        A4 --> A6
    end

    subgraph "Real-Time Inference Pipeline"
        B1["Live Video Input"]:::input
        B2["MediaPipe Keypoint Extraction"]:::process
        B3["Data Preprocessing"]:::data
        B4["Real-Time Prediction (Using action.h5)"]:::ml
        B1 --> B2
        B2 --> B3
        B3 --> B4
    end

    A6 -->|"model_used"| B4

    AR["Auxiliary Resources (ressource, README.md)"]:::aux

    click A4 "https://github.com/oriartyz/signactioninterpreter/blob/main/Action Detection.ipynb"
    click A3 "https://github.com/oriartyz/signactioninterpreter/tree/main/MP_Data"
    click A5 "https://github.com/oriartyz/signactioninterpreter/tree/main/Logs/train"
    click A6 "https://github.com/oriartyz/signactioninterpreter/blob/main/action.h5"
    click AR "https://github.com/oriartyz/signactioninterpreter/tree/main/ressource"

    classDef input fill:#AED6F1,stroke:#21618C,stroke-width:2px;
    classDef process fill:#A9DFBF,stroke:#145A32,stroke-width:2px;
    classDef data fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef ml fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef log fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef model fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef aux fill:#D5F5E3,stroke:#239B56,stroke-width:2px;
```
