How reddit-archive/reddit works
Main Components
a. Web Layer:
• Frontend servers (handling templates in r2/r2/templates)
• API servers (controllers in r2/r2/controllers)
• Load balancers (implied by scale)
b. Application Layer:
• Core business logic (r2/r2/models)
• Authentication services (r2/r2/lib/providers/auth)
• Media handling (r2/r2/lib/providers/media)
• Search functionality (r2/r2/lib/providers/search)
• Cache layer (r2/r2/lib/cache.py)
c. Data Storage:
• PostgreSQL (setup_postgres.sh)
• Cassandra (setup_cassandra.sh)
• Zookeeper (setup_zookeeper.sh)
• RabbitMQ (setup_rabbitmq.sh)
d. Job Processing:
• Queue consumers (upstart/reddit-consumer-*)
• Scheduled jobs (upstart/reddit-job-*)
Key Relationships to Highlight
a. Data Flow:
• Client requests → Load Balancer → Web Servers
• Web Servers → Application Servers
• Application Servers → Cache Layer → Databases
• Event Collection → Message Queues → Consumers
b. Service Dependencies:
• Authentication flow
• Media storage and delivery
• Search indexing and queries
• Cache invalidation paths
Architectural Patterns to Emphasize
a. Message Queue Architecture:
• RabbitMQ consumers for different tasks
• Async processing paths
b. Caching Strategy:
• Multiple cache layers
• Cache invalidation flow
c. Service Oriented Components:
• Auth providers
• Media providers
• Search providers
• Email providers
Implementation Details
a. Key Technologies:
• Python web framework
• PostgreSQL/Cassandra databases
• Memcached/McRouter caching
• RabbitMQ message queue
• Zookeeper for coordination
b. Critical Services:
• Vote processing
• Comment tree building
• Search indexing
• Media handling
• Authentication