# chinmayagrawal775/pixelriver architecture

> Architecture diagram of the GitHub repository chinmayagrawal775/pixelriver, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chinmayagrawal775/pixelriver
- Repository: https://github.com/chinmayagrawal775/pixelriver

## Overview

##### And Represent the Main Components
a. API Server (Node.js + TypeScript)
- This component handles HTTP requests for uploads and status checks.

b. Image Processing Service (Python)
- This worker service consumes messages from Kafka and processes the images (compressing them) and also triggers webhooks.

c. Database (MongoDB)
- Used to store upload metadata and processed result information.

d. Message Queue (Kafka)
- Used to reliably deliver messages from the API service to the image processing service.

e. Caching/Rate-limiting & Pub/Sub System (Redis)
- Redis is used for caching status checks, implementing rate limiting (using a fixed window technique), and handling pub/sub for webhook firing.
- A “Redis” component.

f. Storage (GCP Storage Bucket)
- Files are uploaded and stored in a GCP bucket.
- “GCP Storage” as another external service.

g. Load Balancer (Nginx)
- Although used primarily for production, Nginx is set up as the load balancer and reverse proxy for the API service.
- An “Nginx / LB” component at the edge.

h. Logging & Monitoring
- Winston is used for logging across the services.

##### Relationships and Interactions
- Clients send file uploads and requests to the “Nginx / Load Balancer”, which forwards them to the “API Server”.
- The “API Server” processes the request, stores metadata in “MongoDB”, uploads files to “GCP Storage”, and pushes messages to “Kafka” for asynchronous processing.
- The “Image Processing Worker” (Python service) subscribes to Kafka messages, processes the image, updates the status/results in “MongoDB” (or via cache), and fires relevant webhooks using Redis Pub/Sub.
- “Redis” is integrated with the API for handling rate limits and caching, and also serves as a message broadcaster for webhooks.
- “Logging & Monitoring” is collecting logs from various components (using Winston in Node.js and Python logging in the worker).

##### Architectural Patterns and Design Principles
- Monolithic structure: Although components are logically separated, the system resides in a single repository with segregated directories (api, image_processor, pixelriver docs).
- Separation of concerns: Clear distinctions between API services, image processing tasks, and infrastructure configuration.
- Asynchronous processing: Uses Kafka to decouple the upload request handling from image processing operations.
- Resilience and scalability: Use of caching (Redis) and messaging queues (Kafka) indicate patterns aimed at handling high throughput and ensuring reliable communications.

##### Technologies, Frameworks, and Libraries
- Node.js & TypeScript (for the API)
- Python (for the processing worker)
- MongoDB
- Kafka
- Redis
- GCP (for file storage)
- Nginx
- Winston (for logging)

##### Summary
- Nginx (Load Balancer/Reverse Proxy)
- API Server (Node.js & TypeScript)
- Database (MongoDB)
- Message Queue (Kafka)
- Caching and Rate Limiting (Redis)
- File Storage (GCP Storage Bucket)
- Image Processing Worker (Python-based processing and webhook dispatcher)
- Logging & Monitoring (Winston based and possibly future external monitoring integrations)

## Diagram (Mermaid)

```mermaid
graph TD
    %% External and Client Nodes
    Client("Client"):::client
    Nginx("Nginx / Load Balancer"):::external
    Mongo("MongoDB (Database)"):::external
    Kafka("Kafka (Message Queue)"):::external
    Redis("Redis (Caching & Rate Limiting)"):::external
    GCP("GCP Storage (File Storage)"):::external
    Logger("Logging & Monitoring"):::external

    %% Top-level Flow
    Client -->|"uploads"| Nginx

    %% API Server Subgraph
    subgraph "API Server (Node.js)"
        API_App("app.ts"):::internal
        API_Controller("Controllers\n(upload.ts)"):::internal
        API_Middleware("Middlewares\n(rate-limiting-middleware.ts)"):::internal
        API_Route("Routes\n(upload.ts)"):::internal
        
        API_MongoInfra("Mongo Integration\n(mongo.ts)"):::internal
        API_KafkaInfra("Kafka Integration\n(kafka.ts)"):::internal
        API_RedisInfra("Redis Integration\n(redis.ts)"):::internal
        API_GCPInfra("GCP Integration\n(gcp.ts)"):::internal
        API_LoggerInfra("Logger\n(logger.ts)"):::internal
    end

    %% Worker Service Subgraph
    subgraph "Image Processing Worker (Python)"
        Worker_Main("main.py"):::internal
        Worker_Upload("Upload Processing\n(consumer.py, processor.py)"):::internal
        Worker_Webhook("Webhook Handling\n(main.py, notifier.py, subscriber.py)"):::internal
        
        Worker_Mongo("Mongo Integration\n(mongo.py)"):::internal
        Worker_Kafka("Kafka Integration\n(kafka.py)"):::internal
        Worker_Redis("Redis Integration\n(redis.py)"):::internal
        Worker_GCP("GCP Integration\n(gcp.py)"):::internal
        Worker_Logger("Logger\n(logger.py)"):::internal
    end

    %% Connections from Nginx to API
    Nginx -->|"forwards"| API_App

    %% Internal API connections
    API_App -->|"routes to"| API_Controller
    API_App -->|"applies"| API_Middleware
    API_App -->|"defines"| API_Route
    API_App --> API_MongoInfra
    API_App --> API_KafkaInfra
    API_App --> API_RedisInfra
    API_App --> API_GCPInfra
    API_App --> API_LoggerInfra

    %% API External Integrations
    API_MongoInfra -->|"stores/reads"| Mongo
    API_KafkaInfra -->|"publishes"| Kafka
    API_RedisInfra -->|"rate_limit/cache"| Redis
    API_GCPInfra -->|"uploads"| GCP
    API_LoggerInfra -->|"logs"| Logger

    %% Kafka delivers messages to Worker
    Kafka -->|"delivers"| Worker_Main

    %% Internal Worker connections
    Worker_Main -->|"triggers"| Worker_Upload
    Worker_Main -->|"initiates"| Worker_Webhook
    Worker_Main --> Worker_Mongo
    Worker_Main --> Worker_Redis
    Worker_Main --> Worker_Logger
    Worker_Main --> Worker_GCP

    %% Worker External Integrations
    Worker_Mongo -->|"updates"| Mongo
    Worker_Redis -->|"fires Pub/Sub"| Redis
    Worker_Logger -->|"logs"| Logger
    Worker_GCP -->|"file operations"| GCP

    %% Class Definitions
    classDef internal fill:#D0E1F9,stroke:#1A4D9B,stroke-width:2px;
    classDef external fill:#F9E0D0,stroke:#B34700,stroke-width:2px;
    classDef client fill:#E0F9D0,stroke:#389E0D,stroke-width:2px;

    %% Click Events for API Server Components
    click API_App "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/app.ts"
    click API_Controller "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/http/controllers/upload.ts"
    click API_Middleware "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/http/middlewares/rate-limiting-middleware.ts"
    click API_Route "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/http/routes/upload.ts"
    click API_MongoInfra "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/infra/mongo.ts"
    click API_KafkaInfra "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/infra/kafka.ts"
    click API_RedisInfra "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/infra/redis.ts"
    click API_GCPInfra "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/infra/gcp.ts"
    click API_LoggerInfra "https://github.com/chinmayagrawal775/pixelriver/blob/main/api/src/infra/logger.ts"

    %% Click Events for Worker Service Components
    click Worker_Main "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/main.py"
    click Worker_Upload "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/upload/consumer.py"
    click Worker_Webhook "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/webhook/main.py"
    click Worker_Mongo "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/core/mongo.py"
    click Worker_Kafka "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/core/kafka.py"
    click Worker_Redis "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/core/redis.py"
    click Worker_GCP "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/core/gcp.py"
    click Worker_Logger "https://github.com/chinmayagrawal775/pixelriver/blob/main/image_processor/image_processor/core/logger.py"
```
