How chinmayagrawal775/pixelriver works
And Represent the Main Components
a. API Server (Node.js + TypeScript)
• This component handles HTTP requests for uploads and status checks.
b. Image Processing Service (Python)
• This worker service consumes messages from Kafka and processes the images (compressing them) and also triggers webhooks.
c. Database (MongoDB)
• Used to store upload metadata and processed result information.
d. Message Queue (Kafka)
• Used to reliably deliver messages from the API service to the image processing service.
e. Caching/Rate-limiting & Pub/Sub System (Redis)
• Redis is used for caching status checks, implementing rate limiting (using a fixed window technique), and handling pub/sub for webhook firing.
• A “Redis” component.
f. Storage (GCP Storage Bucket)
• Files are uploaded and stored in a GCP bucket.
• “GCP Storage” as another external service.
g. Load Balancer (Nginx)
• Although used primarily for production, Nginx is set up as the load balancer and reverse proxy for the API service.
• An “Nginx / LB” component at the edge.
h. Logging & Monitoring
• Winston is used for logging across the services.
Relationships and Interactions
• Clients send file uploads and requests to the “Nginx / Load Balancer”, which forwards them to the “API Server”.
• The “API Server” processes the request, stores metadata in “MongoDB”, uploads files to “GCP Storage”, and pushes messages to “Kafka” for asynchronous processing.
• The “Image Processing Worker” (Python service) subscribes to Kafka messages, processes the image, updates the status/results in “MongoDB” (or via cache), and fires relevant webhooks using Redis Pub/Sub.
• “Redis” is integrated with the API for handling rate limits and caching, and also serves as a message broadcaster for webhooks.
• “Logging & Monitoring” is collecting logs from various components (using Winston in Node.js and Python logging in the worker).
Architectural Patterns and Design Principles
• Monolithic structure: Although components are logically separated, the system resides in a single repository with segregated directories (api, image_processor, pixelriver docs).
• Separation of concerns: Clear distinctions between API services, image processing tasks, and infrastructure configuration.
• Asynchronous processing: Uses Kafka to decouple the upload request handling from image processing operations.
• Resilience and scalability: Use of caching (Redis) and messaging queues (Kafka) indicate patterns aimed at handling high throughput and ensuring reliable communications.
Technologies, Frameworks, and Libraries
• Node.js & TypeScript (for the API)
• Python (for the processing worker)
• MongoDB
• Kafka
• Redis
• GCP (for file storage)
• Nginx
• Winston (for logging)
Summary
• Nginx (Load Balancer/Reverse Proxy)
• API Server (Node.js & TypeScript)
• Database (MongoDB)
• Message Queue (Kafka)
• Caching and Rate Limiting (Redis)
• File Storage (GCP Storage Bucket)
• Image Processing Worker (Python-based processing and webhook dispatcher)
• Logging & Monitoring (Winston based and possibly future external monitoring integrations)