How andreaswt/ai-video-sentiment-model works
Project Type and Purpose
• This is a machine learning project focused on multimodal AI, which accepts video inputs and outputs sentiment and emotion predictions.
• It encompasses both training (model development and evaluation) and deployment (inference via AWS SageMaker endpoints) aspects, while also integrating with a SaaS front-end (built using Next.js, React, Tailwind CSS, and Auth.js) that isn’t directly in the repository but is described as part of the complete solution.
File Structure
• Top-level directories include:
• dataset: where the MELD dataset is stored.
• deployment: containing scripts and configurations for deploying the model as an AWS SageMaker endpoint (e.g., deploy_endpoint.py, inference.py, models.py, and a requirements file).
• training: containing scripts for training the model (e.g., train.py, count_parameters.py, melding the dataset, model definitions, logging tests, and installation scripts for dependencies like ffmpeg).
• The separate train_sagemaker.py file, which likely serves as an entry point to initiate training via AWS SageMaker.
• This organization points to a clear separation between training and deployment environments, with a dedicated dataset directory and distinct pipeline scripts for model development.
README for Additional Insights
• The README explains the technical stack and core features: video frame extraction, audio processing, text embedding using BERT, multimodal fusion, emotion & sentiment classification, and TensorBoard logging.
• There is extensive AWS integration: using S3 for dataset storage, SageMaker for training jobs and endpoint deployment, and cloud-based logging (CloudWatch, TensorBoard via S3 sync).
• The instructions also mention building a SaaS with Next.js, React, and related libraries to wrap this model for user interaction, API key management, and quota tracking.
Main Components
a. Core Components:
• Data Layer:
• Dataset Directory → MELD Dataset (ingestion and storage)
• AWS S3: Used for both dataset storage and storing TensorBoard logs.
• Training Pipeline:
• Training Scripts (in the “training” directory and train_sagemaker.py) that handle model training, logging, and possibly pre-processing steps.
• PyTorch and related libraries for building and training the multimodal model.
• SageMaker integration for scalable training.
• Deployment Pipeline:
• Deployment Scripts (in the “deployment” directory, including deploy_endpoint.py and inference.py).
• AWS SageMaker Endpoints for hosting the model.
• IAM roles and policies guiding secure access for deployments.
• SaaS/API Integration:
• Frontend components (though not in the repo, they are described in the README): Built with Next.js, React, Tailwind CSS, Auth.js.
• API Gateway/Server component that manages user requests, invokes the deployed model, and handles API key and quota management.
b. Relationships and Data Flow:
• The dataset flows into the training pipeline where the model is created and evaluated.
• Upon training completion, the trained model is uploaded to AWS S3 and is then deployed as an inference endpoint on SageMaker.
• The SaaS/API layer (frontend and supporting backend not included in the file tree) sends video data to the deployed model, and results are returned to the user.
• TensorBoard logs from training are synchronized from S3 for monitoring.
Practical Steps for the Principal Software Engineer
• Annotations where necessary (e.g., “Uses PyTorch for multimodal fusion”, “Integrated with AWS for scalability and logging”).