# reddit-archive/reddit architecture

> Architecture diagram of the GitHub repository reddit-archive/reddit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reddit-archive/reddit
- Repository: https://github.com/reddit-archive/reddit

## Overview

##### Main Components

a. Web Layer:
- Frontend servers (handling templates in r2/r2/templates)
- API servers (controllers in r2/r2/controllers)
- Load balancers (implied by scale)

b. Application Layer:
- Core business logic (r2/r2/models)
- Authentication services (r2/r2/lib/providers/auth)
- Media handling (r2/r2/lib/providers/media)
- Search functionality (r2/r2/lib/providers/search)
- Cache layer (r2/r2/lib/cache.py)

c. Data Storage:
- PostgreSQL (setup_postgres.sh)
- Cassandra (setup_cassandra.sh)
- Zookeeper (setup_zookeeper.sh)
- RabbitMQ (setup_rabbitmq.sh)

d. Job Processing:
- Queue consumers (upstart/reddit-consumer-*)
- Scheduled jobs (upstart/reddit-job-*)

##### Key Relationships to Highlight

a. Data Flow:
- Client requests → Load Balancer → Web Servers
- Web Servers → Application Servers
- Application Servers → Cache Layer → Databases
- Event Collection → Message Queues → Consumers

b. Service Dependencies:
- Authentication flow
- Media storage and delivery
- Search indexing and queries
- Cache invalidation paths

##### Architectural Patterns to Emphasize

a. Message Queue Architecture:
- RabbitMQ consumers for different tasks
- Async processing paths

b. Caching Strategy:
- Multiple cache layers
- Cache invalidation flow

c. Service Oriented Components:
- Auth providers
- Media providers
- Search providers
- Email providers

##### Implementation Details

a. Key Technologies:
- Python web framework
- PostgreSQL/Cassandra databases
- Memcached/McRouter caching
- RabbitMQ message queue
- Zookeeper for coordination

b. Critical Services:
- Vote processing
- Comment tree building
- Search indexing
- Media handling
- Authentication

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef web fill:#2374f7,color:white
    classDef data fill:#2ecc71,color:white
    classDef cache fill:#f1c40f,color:white
    classDef critical fill:#e74c3c,color:white
    classDef external fill:#95a5a6,color:white
    classDef queue fill:#9b59b6,color:white
    
    %% Client Layer
    Client[Client Requests]

    %% Web Layer
    subgraph WebLayer[Web Layer]
        LB[Load Balancer]
        FE[Frontend Templates]:::web
        API[API Controllers]:::web
    end

    %% Application Layer
    subgraph AppLayer[Application Layer]
        CoreLogic[Core Business Logic]:::web
        AuthService[Authentication Services]:::critical
        MediaService[Media Services]:::web
        SearchService[Search Services]:::web
        ConfigMgmt[Configuration Management]:::web
        TrafficAnalytics[Traffic Analytics]:::web
    end

    %% Cache Layer
    subgraph CacheLayer[Cache Layer]
        Cache[Cache System]:::cache
        McRouter[McRouter]:::cache
    end

    %% Message Queue Layer
    subgraph QueueLayer[Message Queue System]
        RabbitMQ[RabbitMQ]:::queue
        EventCollector[Event Collector]:::queue
        MQHandler[Message Queue Handler]:::queue
    end

    %% Data Storage Layer
    subgraph DataLayer[Data Storage]
        Postgres[(PostgreSQL)]:::data
        Cassandra[(Cassandra)]:::data
        Zookeeper[(Zookeeper)]:::data
        DBOps[Database Operations]:::data
    end

    %% Background Processing
    subgraph Workers[Background Workers]
        QueueConsumers[Queue Consumers]
        ScheduledJobs[Scheduled Jobs]
    end

    %% Relationships
    Client --> LB
    LB --> FE
    LB --> API
    FE --> CoreLogic
    API --> CoreLogic
    CoreLogic --> AuthService
    CoreLogic --> MediaService
    CoreLogic --> SearchService
    CoreLogic --> Cache
    Cache --> McRouter
    McRouter --> Postgres
    McRouter --> Cassandra
    CoreLogic --> EventCollector
    EventCollector --> RabbitMQ
    RabbitMQ --> QueueConsumers
    MQHandler --> RabbitMQ
    ScheduledJobs --> CoreLogic
    DBOps --> Postgres
    DBOps --> Cassandra
    Zookeeper --> CoreLogic
    TrafficAnalytics --> CoreLogic
    ConfigMgmt --> CoreLogic

    %% Click events for component mapping
    click FE "https://github.com/reddit-archive/reddit/tree/master/r2/r2/templates"
    click API "https://github.com/reddit-archive/reddit/tree/master/r2/r2/controllers"
    click CoreLogic "https://github.com/reddit-archive/reddit/tree/master/r2/r2/models"
    click AuthService "https://github.com/reddit-archive/reddit/tree/master/r2/r2/lib/providers/auth"
    click MediaService "https://github.com/reddit-archive/reddit/tree/master/r2/r2/lib/providers/media"
    click SearchService "https://github.com/reddit-archive/reddit/tree/master/r2/r2/lib/providers/search"
    click Cache "https://github.com/reddit-archive/reddit/blob/master/r2/r2/lib/cache.py"
    click Postgres "https://github.com/reddit-archive/reddit/blob/master/install/setup_postgres.sh"
    click Cassandra "https://github.com/reddit-archive/reddit/blob/master/install/setup_cassandra.sh"
    click Zookeeper "https://github.com/reddit-archive/reddit/blob/master/install/setup_zookeeper.sh"
    click RabbitMQ "https://github.com/reddit-archive/reddit/blob/master/install/setup_rabbitmq.sh"
    click QueueConsumers "https://github.com/reddit-archive/reddit/tree/master/upstart/reddit-consumer-*"
    click ScheduledJobs "https://github.com/reddit-archive/reddit/tree/master/upstart/reddit-job-*"
    click EventCollector "https://github.com/reddit-archive/reddit/blob/master/r2/r2/lib/eventcollector.py"
    click MQHandler "https://github.com/reddit-archive/reddit/blob/master/r2/r2/lib/amqp.py"
    click TrafficAnalytics "https://github.com/reddit-archive/reddit/tree/master/scripts/traffic"
    click ConfigMgmt "https://github.com/reddit-archive/reddit/tree/master/r2/r2/config"
    click DBOps "https://github.com/reddit-archive/reddit/tree/master/r2/r2/lib/db"
```
