How aws-samples/video-semantic-search-with-aws-ai-ml-services works
Main Components
• Frontend Web Application:
– Located in the “frontend” directory, built as a React/TypeScript application (note the files like app.tsx, main.tsx, and vite configuration files).
– It serves as the static website (hosted on Amazon S3 and delivered via CloudFront per the README) and handles customer login through Amazon Cognito.
• API & Backend Services:
– The “infrastructure” folder contains SAM templates, Step Functions (step_function.json), and Lambda serverless functions under “infrastructure/functions”. Each subdirectory (e.g., create_job, generate_shot_desc, transcribe, etc.) represents a separate Lambda function that performs a distinct role in the video indexing workflow.
– There is also usage of AWS Step Functions to orchestrate the workflow and integration with Amazon SQS for queuing video indexing requests.
• AWS AI/ML and Media Processing Services:
– Services mentioned in the README include Amazon Rekognition (for detecting scenes, celebrities, and objects), Amazon Transcribe (for generating video transcriptions), Amazon Bedrock (for multimodal analysis and image embedding tasks), and Amazon OpenSearch (used as a vector database for storing embeddings and metadata).
• Supporting Infrastructure:
– The project integrates additional AWS services such as API Gateway (triggering Lambda functions via pre-signed URLs and events), Amazon DynamoDB (storing job metadata and status), and infrastructure layers for components like ffmpeg and opensearch.
• External Dependencies & User Interactions:
– User uploads are managed with S3 pre-signed URLs, and after upload, the rest of the processing is handled by the serverless back end.
Mapping Relationships and Data Flows
• Flow starting with the user uploading a video on the web application.
– Label “Frontend (React App)” connecting to “Amazon S3” (for hosting the static website and storing video files).
• How an S3 event (or an API call via pre-signed URL) triggers API Gateway which in turn fires a Lambda function.
• Next, indicate the queuing of a new video indexing job into Amazon SQS.
– This should branch out into parallel processing steps that include Lambdas for:
o Video shot detection and metadata generation using Amazon Rekognition.
o Transcription through Amazon Transcribe.
o Image processing and embeddings via Amazon Bedrock.
o Generation of shot descriptions and image searches.
• Integration with Amazon Cognito for user management and authentication.
Architectural Patterns & Design Principles
• Serverless and event-driven architecture (using AWS Lambda, SQS, and Step Functions).
• Microservice-like approach where each AWS Lambda function (organized in separate directories) handles a specific aspect of processing.
• Separation of concerns: the frontend is built and deployed independently of the backend services.
• Mention the use of AWS SAM CLI and Docker for local development and deployment, showing build/deployment pipelines (e.g., from SAM template and samconfig.toml).
Relevant Technologies, Frameworks, and Libraries
• For the Frontend: React, TypeScript, Vite, and npm are used.
• For the Serverless Infrastructure: AWS Lambda, Step Functions, SQS, and SAM.
• For Video Processing / AI: Amazon Rekognition, Amazon Transcribe, Amazon Bedrock, and Amazon OpenSearch.
• For Data storage and State management: Amazon DynamoDB and S3.
• Supporting layers (e.g., ffmpeg layer) visible in the “infrastructure/layers” directory.