How sarthakkhandelwal7/rag_data_ingestion_pipline works
Key Components to Include:
External Services Layer
• OpenAI API (for embeddings generation)
• Pinecone Vector Database
AWS Infrastructure Layer
• S3 Bucket (source)
• AWS Parameter Store
• AWS Lambda (containerized)
• Amazon ECR (for Docker image)
Use AWS architectural symbols for these components.
Data Flow and Process Layer
Complete flow of data processing with numbered steps:
1. HTML document upload to S3
2. S3 event trigger
3. Lambda function activation
4. Parameter retrieval from Parameter Store
5. Document processing and embedding generation
6. Vector storage in Pinecone
Component Details
Annotations for key configurations:
• Lambda: 512MB memory, 90s timeout
• Document chunking: 600 chars, 50 char overlap
• Embedding model: text-embedding-3-small
• Docker container specifications
Security Layer
Security aspects:
• API key storage in Parameter Store
• IAM roles and permissions
• S3 bucket access controls
Infrastructure as Code
A separate section showing Terraform management:
• Resource provisioning
• Configuration management
• Environment variable handling
Specific Elements to Include:
Process Flows
Number each step in the pipeline:
1. Document Upload
2. Event Triggering
3. Function Execution
4. Parameter Retrieval
5. Document Processing
6. Embedding Generation
7. Vector Storage
Remember to:
• Use consistent styling
• Version information
• Document all external dependencies
• Error handling paths
• Scaling points