# andreaswt/ai-video-sentiment-model architecture

> Architecture diagram of the GitHub repository andreaswt/ai-video-sentiment-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andreaswt/ai-video-sentiment-model
- Repository: https://github.com/andreaswt/ai-video-sentiment-model

## Overview

##### Project Type and Purpose
- This is a machine learning project focused on multimodal AI, which accepts video inputs and outputs sentiment and emotion predictions.
- It encompasses both training (model development and evaluation) and deployment (inference via AWS SageMaker endpoints) aspects, while also integrating with a SaaS front-end (built using Next.js, React, Tailwind CSS, and Auth.js) that isn’t directly in the repository but is described as part of the complete solution.

##### File Structure
- Top-level directories include:
- dataset: where the MELD dataset is stored.
- deployment: containing scripts and configurations for deploying the model as an AWS SageMaker endpoint (e.g., deploy_endpoint.py, inference.py, models.py, and a requirements file).
- training: containing scripts for training the model (e.g., train.py, count_parameters.py, melding the dataset, model definitions, logging tests, and installation scripts for dependencies like ffmpeg).
- The separate train_sagemaker.py file, which likely serves as an entry point to initiate training via AWS SageMaker.
- This organization points to a clear separation between training and deployment environments, with a dedicated dataset directory and distinct pipeline scripts for model development.

##### README for Additional Insights
- The README explains the technical stack and core features: video frame extraction, audio processing, text embedding using BERT, multimodal fusion, emotion & sentiment classification, and TensorBoard logging.
- There is extensive AWS integration: using S3 for dataset storage, SageMaker for training jobs and endpoint deployment, and cloud-based logging (CloudWatch, TensorBoard via S3 sync).
- The instructions also mention building a SaaS with Next.js, React, and related libraries to wrap this model for user interaction, API key management, and quota tracking.

##### Main Components
a. Core Components:
- Data Layer:
- Dataset Directory → MELD Dataset (ingestion and storage)
- AWS S3: Used for both dataset storage and storing TensorBoard logs.
- Training Pipeline:
- Training Scripts (in the “training” directory and train_sagemaker.py) that handle model training, logging, and possibly pre-processing steps.
- PyTorch and related libraries for building and training the multimodal model.
- SageMaker integration for scalable training.
- Deployment Pipeline:
- Deployment Scripts (in the “deployment” directory, including deploy_endpoint.py and inference.py).
- AWS SageMaker Endpoints for hosting the model.
- IAM roles and policies guiding secure access for deployments.
- SaaS/API Integration:
- Frontend components (though not in the repo, they are described in the README): Built with Next.js, React, Tailwind CSS, Auth.js.
- API Gateway/Server component that manages user requests, invokes the deployed model, and handles API key and quota management.
b. Relationships and Data Flow:
- The dataset flows into the training pipeline where the model is created and evaluated.
- Upon training completion, the trained model is uploaded to AWS S3 and is then deployed as an inference endpoint on SageMaker.
- The SaaS/API layer (frontend and supporting backend not included in the file tree) sends video data to the deployed model, and results are returned to the user.
- TensorBoard logs from training are synchronized from S3 for monitoring.

##### Practical Steps for the Principal Software Engineer
- Annotations where necessary (e.g., “Uses PyTorch for multimodal fusion”, “Integrated with AWS for scalability and logging”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        D1("Dataset (MELD)"):::data
        D2("AWS S3"):::aws
    end

    %% ML Pipeline
    subgraph "ML Pipeline"
        %% Training Pipeline
        subgraph "Training Pipeline"
            T3("Data Preparation: meld_dataset.py"):::training
            T1("Training: train.py"):::training
            T2("Parameter Analysis: count_parameters.py"):::training
            T5("Logging Tests: test_logging.py"):::training
            T4("Training Model Definitions: models.py"):::training
            T6("FFmpeg Installation: install_ffmpeg.py"):::training
            T7("SageMaker Training Entry: train_sagemaker.py"):::aws
        end

        %% Deployment Pipeline
        subgraph "Deployment Pipeline"
            Dp1("Endpoint Deployment: deploy_endpoint.py"):::deployment
            Dp2("Inference Script: inference.py"):::deployment
            Dp3("Deployment Model Definitions: models.py"):::deployment
            Dp4("Deployment Requirements: requirements.txt"):::deployment
            Dp5("AWS SageMaker Endpoint"):::aws
        end
    end

    %% SaaS/API Integration
    subgraph "SaaS/API Integration"
        S1("Frontend/API (Next.js/React)"):::saas
        S2("API Gateway/Server"):::saas
        U("User"):::user
    end

    %% Data Flow Connections
    D1 -->|"data_to_preprocessing"| T3
    T3 -->|"preprocess_to_train"| T1
    T1 -->|"check_parameters"| T2
    T1 -->|"logging"| T5
    T1 -->|"upload_model_and_logs"| D2
    T1 -->|"invoke_SageMaker_train"| T7
    T7 -->|"initiate_deploy"| Dp1
    Dp1 -->|"deploy_model"| Dp5
    Dp2 -->|"wrap_inference"| Dp5
    Dp5 -->|"serve_prediction"| S2
    S2 -->|"api_call"| S1
    S1 -->|"user_request"| U

    %% Click Events
    click D1 "https://github.com/andreaswt/ai-video-sentiment-model/tree/main/dataset/"
    click T1 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/training/train.py"
    click T2 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/training/count_parameters.py"
    click T3 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/training/meld_dataset.py"
    click T4 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/training/models.py"
    click T5 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/training/test_logging.py"
    click T6 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/training/install_ffmpeg.py"
    click T7 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/train_sagemaker.py"
    click Dp1 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/deployment/deploy_endpoint.py"
    click Dp2 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/deployment/inference.py"
    click Dp3 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/deployment/models.py"
    click Dp4 "https://github.com/andreaswt/ai-video-sentiment-model/blob/main/deployment/requirements.txt"

    %% Styles
    classDef data fill:#FFD1DC,stroke:#333,stroke-width:2px;
    classDef aws fill:#C6E2FF,stroke:#333,stroke-width:2px;
    classDef training fill:#C1FFC1,stroke:#333,stroke-width:2px;
    classDef deployment fill:#FFCCE5,stroke:#333,stroke-width:2px;
    classDef saas fill:#FFF6A5,stroke:#333,stroke-width:2px;
    classDef user fill:#CCF9FF,stroke:#333,stroke-width:2px;
```
