# bayajidalam/r-queue architecture

> Architecture diagram of the GitHub repository bayajidalam/r-queue, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bayajidalam/r-queue
- Repository: https://github.com/bayajidalam/r-queue

## Overview

##### Main Components

a) Client Layer:
- React Frontend Application
- Dashboard Components
- Job Input Interface
- Real-time Monitoring UI

b) Server Layer:
- Express.js Backend API
- Job Queue Management System
- Worker Pool Management
- Circuit Breaker Implementation
- Health Check System

c) Redis Cluster Layer:
- Multiple Redis Nodes (Primary-Replica Setup)
- Job Queue Storage
- Job Status Management
- Dead Letter Queue

d) Infrastructure Layer:
- AWS VPC
- Public/Private Subnets
- EC2 Instances
- Security Groups
- NAT Gateway

**[Client Tier]**
- ↕
**[API/Server Tier]**
- ↕
**[Redis Cluster Tier]**
- ↕
**[Infrastructure Tier]**

##### Component Relationships to Highlight

a) Client-Server Communication:
- REST API calls between frontend and backend
- WebSocket connections for real-time updates
- Health check endpoints

b) Server-Redis Interaction:
- Job queue operations
- Worker node communication
- Cluster management
- State persistence

c) Infrastructure Relationships:
- Network flow between components
- Security group associations
- Load balancer distributions

##### Special Elements

a) Job Flow Visualization:
**[Job Input] → [Queue] → [Worker Pool] → [Results]**
- ↓
- [Dead Letter Queue]

b) High Availability Features:
- Redis cluster replication
- Multiple worker nodes
- Circuit breaker pattern

c) Scaling Components:
- Auto-scaling groups
- Load balancers
- Worker pool scaling

##### Additional Details

- Metrics collection points
- Retry mechanisms
- Job priority handling
- Health check paths
- Monitoring points

##### Implementation Notes

- Rectangles: Services and components
- Clouds: AWS services
- Hexagons: Security groups

##### Key Aspects to Emphasize

- Fault tolerance mechanisms
- Scalability points
- Security boundaries
- Data persistence
- Monitoring capabilities
- Worker node management

Remember to maintain a balance between detail and clarity, focusing on the distributed nature of the system while showing how the components work together to provide a reliable job queue system.

- Job submission and distribution
- Worker node scaling
- Redis cluster operations
- Infrastructure management
- Monitoring and health checks
- Security and network isolation

This will provide a comprehensive view of the system while maintaining readability and understanding of the complex distributed architecture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Define styles
    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef backend fill:#2ecc71,stroke:#27ae60,color:white
    classDef redis fill:#e74c3c,stroke:#c0392b,color:white
    classDef infra fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef security fill:#f1c40f,stroke:#f39c12,color:black

    subgraph ClientLayer["Client Layer"]
        Dashboard["Dashboard UI"]:::frontend
        JobInput["Job Input Interface"]:::frontend
        MonitoringUI["Real-time Monitoring"]:::frontend
    end

    subgraph ServerLayer["Server Layer"]
        API["Express Backend API"]:::backend
        JobQueue["Job Queue Manager"]:::backend
        WorkerPool["Worker Pool"]:::backend
        CircuitBreaker["Circuit Breaker"]:::backend
        HealthCheck["Health Check System"]:::backend
        MetricsSystem["Metrics System"]:::backend
    end

    subgraph RedisLayer["Redis Cluster Layer"]
        Primary1["Primary Node 1"]:::redis
        Primary2["Primary Node 2"]:::redis
        Primary3["Primary Node 3"]:::redis
        Replica1["Replica Node 1"]:::redis
        Replica2["Replica Node 2"]:::redis
        Replica3["Replica Node 3"]:::redis
        DLQ["Dead Letter Queue"]:::redis
    end

    subgraph InfraLayer["AWS Infrastructure"]
        VPC["VPC"]:::infra
        PublicSubnet["Public Subnet"]:::infra
        PrivateSubnet["Private Subnet"]:::infra
        NAT["NAT Gateway"]:::infra
        SecGroup["Security Groups"]:::security
    end

    %% Relationships
    Dashboard --> API
    JobInput --> API
    MonitoringUI --> API
    API --> JobQueue
    JobQueue --> WorkerPool
    WorkerPool --> Primary1
    WorkerPool --> Primary2
    WorkerPool --> Primary3
    Primary1 --> Replica1
    Primary2 --> Replica2
    Primary3 --> Replica3
    JobQueue --> DLQ
    API --> HealthCheck
    API --> MetricsSystem
    CircuitBreaker --> Primary1
    CircuitBreaker --> Primary2
    CircuitBreaker --> Primary3

    %% Infrastructure Connections
    VPC --> PublicSubnet
    VPC --> PrivateSubnet
    PublicSubnet --> NAT
    NAT --> PrivateSubnet
    SecGroup --> PublicSubnet
    SecGroup --> PrivateSubnet

    %% Click Events
    click Dashboard "https://github.com/BayajidAlam/r-queue/blob/main/client/src/Components/JobQueueDashboard/JobQueueDashboard.tsx"
    click JobInput "https://github.com/BayajidAlam/r-queue/blob/main/client/src/Components/InputJob/JobSimulationInputPage.tsx"
    click API "https://github.com/BayajidAlam/r-queue/tree/main/server/src/api/routes"
    click HealthCheck "https://github.com/BayajidAlam/r-queue/blob/main/server/src/api/routes/health.ts"
    click JobQueue "https://github.com/BayajidAlam/r-queue/blob/main/server/src/services/JobQueue.ts"
    click WorkerPool "https://github.com/BayajidAlam/r-queue/blob/main/server/src/workers/Worker.ts"
    click CircuitBreaker "https://github.com/BayajidAlam/r-queue/blob/main/server/src/utils/CircuitBreaker.ts"
    click Primary1 "https://github.com/BayajidAlam/r-queue/blob/main/server/src/config/redis.ts"
    click VPC "https://github.com/BayajidAlam/r-queue/blob/main/IaC/pulumi/index.ts"
    click Primary1 "https://github.com/BayajidAlam/r-queue/blob/main/IaC/ansible/playbooks/redis-cluster.yml"
    click API "https://github.com/BayajidAlam/r-queue/blob/main/IaC/ansible/playbooks/backend-setup.yml"
    click Dashboard "https://github.com/BayajidAlam/r-queue/blob/main/IaC/ansible/playbooks/frontend-setup.yml"
    click MetricsSystem "https://github.com/BayajidAlam/r-queue/blob/main/server/src/api/routes/metrics.ts"
    click JobQueue "https://github.com/BayajidAlam/r-queue/blob/main/server/src/api/routes/jobs.ts"
    click Primary1 "https://github.com/BayajidAlam/r-queue/blob/main/IaC/ansible/templates/redis.conf.j2"
```
