How dropbox/hackpad works
Project Type and Purpose
• Hackpad is a full-stack, real-time collaborative wiki built on top of the open-source Etherpad editor.
• It comprises multiple clients (web browser, iOS app, macOS app), a Node.js based real-time server, supporting infrastructure services in Java/Scala, databases (MySQL, Solr), caches, static asset servers, and external integrations (OAuth providers, Dropbox, APNs, Stripe, email, analytics).
High-Level Components
A. Clients
• Web browser + Static assets (HTML/CSS/JS served by Node.js or CDN)
• Native iOS app (Objective-C code under client/ios/Hackpad, HackpadKit for sync)
• Native macOS app (Objective-C code under client/osx/Hackpad)
B. Load Balancer & Reverse Proxy
• NGINX (contrib/nginx.conf) or Docker routed entrypoints (bin/docker-entrypoint.sh)
C. API & Real-Time Servers
• Node.js/Etherpad server (etherpad/src/main.js + control/, collab/ modules)
• Java/Scala OpenOffice conversion microservice (infrastructure/com.etherpad.openofficeservice)
D. Data Stores & Search
• MySQL (pad content, users, migrations under etherpad/src/etherpad/db_migrations)
• Solr (full-text search, under etherpad/solr)
E. Caching & Message Layers
• Memcached (s3 cache for assets, under infrastructure/lib)
• In-process pub/sub for collaboration (Etherpad’s easysync2)
F. Static Assets & Themes
• Static file server or CDN (etherpad/src/static, themes under src/themes)
• Asset pipeline tools (YUICOMPRESSOR, LESS under infrastructure/yuicompressor)
G. Background & Scheduled Jobs
• Cron scripts (contrib/cron) for cleanup, migrations
• Runit or Docker “run.sh” wrappers (contrib/runit, bin/run.sh)
H. External Integrations
• OAuth2 / Facebook SDK / Google SDK
• Dropbox sync (importexport/dropbox.js)
• APNs (java-apns)
• Stripe (billing tables, pro features)
• Email gateway, Mixpanel analytics
Relationships and Data Flows
• Web & native clients ←→ Load Balancer / Proxy via HTTPS
• Static assets (CSS/JS) served directly
• API calls (REST endpoints under control/api_v1_control.js)
• WebSocket (or long-polling) connections for real-time collab (collab_server.js)
• Node.js server ↔ MySQL for read/write pad data
• Node.js server ↔ Solr via HTTP for indexing/search
• Node.js server ↔ Memcached for session storage and caching
• Node.js server → Background conversion service over internal RPC/HTTP
• Background jobs (cron/runit) → Database maintenance, migrations
• Node.js server → External OAuth providers (Facebook, Google), email & billing services
Architectural Patterns & Principles
• Layered MVC: controllers in control/, models in pad/, helpers under helpers.js
• Event-driven: collaboration over websockets, internal pub/sub, asynchronous I/O
• Modular: Etherpad’s module system (framework-src/modules), plug-in architecture
• Microservice for heavy tasks (OpenOffice conversion in Scala/Java)
• Static asset pipeline: LESS → CSS, YUICompressor → minified JS/CSS
Essential Elements to Capture
• Round-cornered rectangles for each component with a title (e.g., “Etherpad Node.js Cluster”)
• Boundary lines showing trust zones (public internet vs. internal network vs. datacenter)