How abir0/amazon-products-analytics works
Main Components
A. Client Layer:
• Streamlit Frontend (app/frontend/app.py)
• External API Consumers
B. Application Layer:
• FastAPI Backend (app/api/)
• Product Router (products.py)
• RAG Router (rag.py)
• Scheduler Router (scheduler.py)
C. Service Layer:
• Scraping Service (app/scraper/)
• RAG Service (app/rag/)
• Database Service (app/database.py)
D. Data Layer:
• PostgreSQL Database
• Weaviate Vector Database
• S3 Storage (for images)
E. External Services:
• Amazon Website (data source)
• OpenAI API
• Chrome WebDriver
[Client Layer] → [Application Layer] → [Service Layer] → [Data Layer]
• ↑ ↑ ↑ ↑
• ┴──────────────────────┴────────────[External Services]
Detailed Component Breakdown
• Rectangles: Main services
• Hexagons: External services
Key Flows to Highlight
A. Scraping Flow:
[Scheduler] → [Amazon Scraper] → [Amazon Website]
• ↓
[PostgreSQL] ← [Data Processing]
B. RAG Flow:
[User Query] → [RAG Router] → [Chatbot] → [Weaviate]
• ↓ ↓
• [PostgreSQL] [OpenAI API]
C. Product API Flow:
[Client] → [Product Router] → [Database Service] → [PostgreSQL]
Important Details
A. Infrastructure Components:
• Docker containers (show containerized services)
• AWS deployment architecture
• Load balancers (if applicable)
B. Data Flow Specifications:
• API endpoints (/products, /rag, /scheduler)
• Database connections
• Async operations
C. Security Layer:
• API authentication
• Database access
• External API keys
AWS-Specific Components
Since the README mentions AWS deployment, include:
• EC2 instances for API
• ECS for container orchestration
• RDS for PostgreSQL
• SageMaker for AI models
• S3 for storage
• CloudWatch for monitoring
AWS deployment architecture
Remember to include annotations explaining key architectural decisions and any potential bottlenecks or scaling considerations.
This system design should reflect the modern microservices architecture while showing the integration of AI capabilities through the RAG system, making it a comprehensive representation of this analytics platform.