# sarthakkhandelwal7/rag_data_ingestion_pipline architecture

> Architecture diagram of the GitHub repository sarthakkhandelwal7/rag_data_ingestion_pipline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarthakkhandelwal7/rag_data_ingestion_pipline
- Repository: https://github.com/sarthakkhandelwal7/rag_data_ingestion_pipline

## Overview

Key Components to Include:

##### External Services Layer
- OpenAI API (for embeddings generation)
- Pinecone Vector Database

##### AWS Infrastructure Layer
- S3 Bucket (source)
- AWS Parameter Store
- AWS Lambda (containerized)
- Amazon ECR (for Docker image)
Use AWS architectural symbols for these components.

##### Data Flow and Process Layer
Complete flow of data processing with numbered steps:
1. HTML document upload to S3
2. S3 event trigger
3. Lambda function activation
4. Parameter retrieval from Parameter Store
5. Document processing and embedding generation
6. Vector storage in Pinecone

##### Component Details
Annotations for key configurations:
- Lambda: 512MB memory, 90s timeout
- Document chunking: 600 chars, 50 char overlap
- Embedding model: text-embedding-3-small
- Docker container specifications

##### Security Layer
Security aspects:
- API key storage in Parameter Store
- IAM roles and permissions
- S3 bucket access controls

##### Infrastructure as Code
A separate section showing Terraform management:
- Resource provisioning
- Configuration management
- Environment variable handling

Specific Elements to Include:

##### Process Flows
Number each step in the pipeline:
1. Document Upload
2. Event Triggering
3. Function Execution
4. Parameter Retrieval
5. Document Processing
6. Embedding Generation
7. Vector Storage

Remember to:
- Use consistent styling
- Version information
- Document all external dependencies
- Error handling paths
- Scaling points

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styling
    classDef aws fill:#FF9900,stroke:#232F3E,color:white
    classDef external fill:#85B4D0,stroke:#232F3E,color:white
    classDef vector fill:#9D72D8,stroke:#232F3E,color:white
    classDef config fill:#A9A9A9,stroke:#232F3E,color:white
    classDef security fill:#FF6B6B,stroke:#232F3E,color:white
    classDef process fill:#FFFFFF,stroke:#232F3E,color:black

    %% External Services
    OpenAI["OpenAI API\n(text-embedding-3-small)"]:::external
    Pinecone["Pinecone Vector DB"]:::vector

    %% AWS Services
    S3["S3 Bucket\n(Document Storage)"]:::aws
    ParamStore["AWS Parameter Store\n(API Keys)"]:::security
    Lambda["Lambda Function\n(512MB, 90s timeout)"]:::aws
    ECR["Amazon ECR\n(Docker Registry)"]:::aws

    %% Infrastructure Components
    subgraph Infrastructure["Infrastructure as Code"]
        Terraform["Terraform Resources"]:::config
        TFVars["Environment Variables"]:::config
        TFConfig["Configuration"]:::config
    end

    %% Process Components
    subgraph Process["Document Processing"]
        Upload["1. Document Upload"]:::process
        Trigger["2. Event Trigger"]:::process
        Execute["3. Function Execution"]:::process
        Process_Doc["4. Document Processing\n(600 chars, 50 overlap)"]:::process
        Generate["5. Embedding Generation"]:::process
        Store["6. Vector Storage"]:::process
    end

    %% Connections
    S3 --> Trigger
    Trigger --> Lambda
    Lambda --> ParamStore
    Lambda --> OpenAI
    OpenAI --> Lambda
    Lambda --> Pinecone
    ECR --> Lambda

    %% Process Flow
    Upload --> S3
    Trigger --> Execute
    Execute --> Process_Doc
    Process_Doc --> Generate
    Generate --> Store

    %% Infrastructure Management
    Terraform --> S3
    Terraform --> Lambda
    Terraform --> ECR
    TFVars --> Terraform
    TFConfig --> Terraform

    %% Click Events
    click Lambda "https://github.com/sarthakkhandelwal7/RAG_Data_Ingestion_Pipline/blob/event_driven_lambda/lambda_function/lambda_function.py"
    click ECR "https://github.com/sarthakkhandelwal7/RAG_Data_Ingestion_Pipline/tree/event_driven_lambda/lambda_function/Dockerfile"
    click Lambda "https://github.com/sarthakkhandelwal7/RAG_Data_Ingestion_Pipline/tree/event_driven_lambda/lambda_function/Pipfile"
    click Terraform "https://github.com/sarthakkhandelwal7/RAG_Data_Ingestion_Pipline/blob/event_driven_lambda/terraform-pipline/main.tf"
    click TFVars "https://github.com/sarthakkhandelwal7/RAG_Data_Ingestion_Pipline/blob/event_driven_lambda/terraform-pipline/variables.tf"
    click TFConfig "https://github.com/sarthakkhandelwal7/RAG_Data_Ingestion_Pipline/blob/event_driven_lambda/terraform-pipline/tfvars-example.md"
```
