# aws-samples/video-semantic-search-with-aws-ai-ml-services architecture

> Architecture diagram of the GitHub repository aws-samples/video-semantic-search-with-aws-ai-ml-services, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/video-semantic-search-with-aws-ai-ml-services
- Repository: https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services

## Overview

##### Main Components
- Frontend Web Application:
– Located in the “frontend” directory, built as a React/TypeScript application (note the files like app.tsx, main.tsx, and vite configuration files).
– It serves as the static website (hosted on Amazon S3 and delivered via CloudFront per the README) and handles customer login through Amazon Cognito.
- API & Backend Services:
– The “infrastructure” folder contains SAM templates, Step Functions (step_function.json), and Lambda serverless functions under “infrastructure/functions”. Each subdirectory (e.g., create_job, generate_shot_desc, transcribe, etc.) represents a separate Lambda function that performs a distinct role in the video indexing workflow.
– There is also usage of AWS Step Functions to orchestrate the workflow and integration with Amazon SQS for queuing video indexing requests.
- AWS AI/ML and Media Processing Services:
– Services mentioned in the README include Amazon Rekognition (for detecting scenes, celebrities, and objects), Amazon Transcribe (for generating video transcriptions), Amazon Bedrock (for multimodal analysis and image embedding tasks), and Amazon OpenSearch (used as a vector database for storing embeddings and metadata).
- Supporting Infrastructure:
– The project integrates additional AWS services such as API Gateway (triggering Lambda functions via pre-signed URLs and events), Amazon DynamoDB (storing job metadata and status), and infrastructure layers for components like ffmpeg and opensearch.
- External Dependencies & User Interactions:
– User uploads are managed with S3 pre-signed URLs, and after upload, the rest of the processing is handled by the serverless back end.

##### Mapping Relationships and Data Flows
- Flow starting with the user uploading a video on the web application.
– Label “Frontend (React App)” connecting to “Amazon S3” (for hosting the static website and storing video files).
- How an S3 event (or an API call via pre-signed URL) triggers API Gateway which in turn fires a Lambda function.
- Next, indicate the queuing of a new video indexing job into Amazon SQS.
– This should branch out into parallel processing steps that include Lambdas for:
o Video shot detection and metadata generation using Amazon Rekognition.
o Transcription through Amazon Transcribe.
o Image processing and embeddings via Amazon Bedrock.
o Generation of shot descriptions and image searches.
- Integration with Amazon Cognito for user management and authentication.

##### Architectural Patterns & Design Principles
- Serverless and event-driven architecture (using AWS Lambda, SQS, and Step Functions).
- Microservice-like approach where each AWS Lambda function (organized in separate directories) handles a specific aspect of processing.
- Separation of concerns: the frontend is built and deployed independently of the backend services.
- Mention the use of AWS SAM CLI and Docker for local development and deployment, showing build/deployment pipelines (e.g., from SAM template and samconfig.toml).

##### Relevant Technologies, Frameworks, and Libraries
- For the Frontend: React, TypeScript, Vite, and npm are used.
- For the Serverless Infrastructure: AWS Lambda, Step Functions, SQS, and SAM.
- For Video Processing / AI: Amazon Rekognition, Amazon Transcribe, Amazon Bedrock, and Amazon OpenSearch.
- For Data storage and State management: Amazon DynamoDB and S3.
- Supporting layers (e.g., ffmpeg layer) visible in the “infrastructure/layers” directory.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    subgraph "User Interaction"
        U["User"]:::aws
        FE["Frontend (React/TypeScript)"]:::frontend
        COG["Amazon Cognito"]:::aws
    end

    %% Hosting & Storage Layer
    subgraph "Hosting & Storage"
        S3["Amazon S3 (Static Hosting & Video Storage)"]:::storage
        CF["Amazon CloudFront"]:::storage
    end

    %% Backend Processing & Orchestration Layer
    subgraph "Backend Processing & Orchestration"
        AG["API Gateway"]:::aws
        LAMBDA["AWS Lambda Functions"]:::aws
        SQS["Amazon SQS Queue"]:::aws
        SF["AWS Step Functions Workflow"]:::aws
    end

    %% Video Processing & AI Services Layer
    subgraph "Video Processing & AI Services"
        REK["Amazon Rekognition"]:::ai
        TRANS["Amazon Transcribe"]:::ai
        BED["Amazon Bedrock"]:::ai
        OS["Amazon OpenSearch"]:::database
        DDB["Amazon DynamoDB"]:::database
    end

    %% Infrastructure Deployment & Supporting Layers
    subgraph "Infrastructure Deployment & Layers"
        SAMT["SAM Template"]:::infra
        SAMC["SAM Config"]:::infra
        FFMPEG["ffmpeg Layer"]:::infra
        OSLAYER["OpenSearch Layer"]:::infra
    end

    %% Data Flow Connections
    U --> FE
    FE -->|"upload_video"| S3
    FE --> COG
    S3 --> CF
    S3 -->|"trigger"| AG
    AG -->|"invoke"| LAMBDA
    LAMBDA -->|"queue_job"| SQS
    SQS --> SF
    SF -->|"orchestrates"| LAMBDA

    %% Parallel processing via Step Functions
    SF -->|"parallel|rekognition"| REK
    SF -->|"parallel|transcribe"| TRANS
    SF -->|"parallel|embeddings"| BED

    %% Processing outputs
    REK -->|"metadata"| OS
    TRANS -->|"transcript"| DDB
    BED -->|"embeddings"| OS
    LAMBDA -->|"job_status"| DDB

    %% Deployment Connections
    SAMT -.-> LAMBDA
    SAMC -.-> LAMBDA
    LAMBDA -.-> FFMPEG
    LAMBDA -.-> OSLAYER

    %% Detailed AWS Lambda Functions (microservices)
    LAMBDA --> CJ["create_job Lambda"]:::lambda
    LAMBDA --> GSD["generate_shot_desc Lambda"]:::lambda
    LAMBDA --> RCD["rekognition_celebrity_detection Lambda"]:::lambda
    LAMBDA --> RSD["rekognition_shot_detection Lambda"]:::lambda
    LAMBDA --> RSS["rekognition_shot_detection_sns Lambda"]:::lambda
    LAMBDA --> TR["transcribe Lambda"]:::lambda
    LAMBDA --> ET["eventbridge_transcribe Lambda"]:::lambda
    LAMBDA --> GSI["generate_shot_image Lambda"]:::lambda
    LAMBDA --> EA["embedding_aoss Lambda"]:::lambda
    LAMBDA --> GA["get_all_jobs Lambda"]:::lambda
    LAMBDA --> PV["presignedurl_video Lambda"]:::lambda
    LAMBDA --> SFN["stepfunction orchestration Lambda"]:::lambda
    LAMBDA --> FJ["failedjob Lambda"]:::lambda
    LAMBDA --> CO["completedjob Lambda"]:::lambda
    LAMBDA --> RO["rekognize_other_figures Lambda"]:::lambda
    LAMBDA --> SL["search Lambda"]:::lambda

    %% Click Events for Component Mappings
    click FE "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/frontend/"
    click CJ "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/create_job/"
    click GSD "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/generate_shot_desc/"
    click RCD "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/rekognition_celebrity_detection/"
    click RSD "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/rekognition_shot_detection/"
    click RSS "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/rekognition_shot_detection_sns/"
    click TR "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/transcribe/"
    click ET "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/eventbridge_transcribe/"
    click GSI "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/generate_shot_image/"
    click EA "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/embedding_aoss/"
    click GA "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/get_all_jobs/"
    click PV "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/presignedurl_video/"
    click SFN "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/stepfunction/"
    click FJ "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/failedjob/"
    click CO "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/completedjob/"
    click RO "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/rekognize_other_figures/"
    click SL "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/functions/search/"
    click SF "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/blob/main/infrastructure/step_function.json"
    click SAMT "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/blob/main/infrastructure/template.yaml"
    click SAMC "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/blob/main/infrastructure/samconfig.toml"
    click FFMPEG "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/layers/ffmpeg/"
    click OSLAYER "https://github.com/aws-samples/video-semantic-search-with-aws-ai-ml-services/tree/main/infrastructure/layers/opensearch/"

    %% Styling
    classDef frontend fill:#b3e5fc,stroke:#0277bd,stroke-width:2px;
    classDef storage fill:#ffe0b2,stroke:#ef6c00,stroke-width:2px;
    classDef aws fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px;
    classDef ai fill:#f8bbd0,stroke:#880e4f,stroke-width:2px;
    classDef database fill:#d1c4e9,stroke:#512da8,stroke-width:2px;
    classDef infra fill:#fff9c4,stroke:#fbc02d,stroke-width:2px;
    classDef lambda fill:#e1bee7,stroke:#6a1b9a,stroke-width:2px;
```
