How youzan/nsq works
Project type and purpose
• This is a fork of NSQ, a realtime distributed messaging platform implemented in Go.
• Purpose: scale‑out message queuing with built‑in replication, high availability (HA) via etcd, topic partitioning, ordered delivery, and a Web UI (nsqadmin).
• Components span multiple Go binaries (nsqd, nsqlookupd, nsqadmin, coordination service), client tools under apps/, and internal libraries.
File structure
Top‑level directories:
• nsqd/: core message queue daemon (disk queue, in-memory queues, protocol handlers, pluggable engine)
• nsqlookupd/: discovery service for topic→nsqd node mappings
• consistence/: etcd‑based coordination (leader election, metadata placement, replication logic)
• nsqadmin/: browser‑based management UI (Go HTTP server + static assets)
• apps/: command‑line utilities (producers, consumers, admin scripts)
• internal/: shared libraries (protocol, HTTP API, auth, logging, metrics, utils)
• bench/: benchmarking tools
• jepsen/: distributed correctness tests
Config files: Dockerfile, Makefile, contrib/*.cfg, Gopkg.toml, Go modules.
Key insights from README
• The fork adds: replication, HA (etcd cluster), auto‑balancing, partitioning, ordered delivery, history consumption, tracing, performance improvements.
• External dependency: etcd cluster for metadata coordination.
• Clients discover topics via nsqlookupd, connect to nsqd over TCP/HTTP.
• nsqadmin interacts with lookupd and nsqd via HTTP APIs.
Main components
a. Clients (Producers/Consumers)
• Custom apps built on go‑nsq/other SDKs.
• CLI tools under apps/ (nsq_pubsub, nsq_tail, etc.).
b. nsqlookupd cluster
• Service registry: maintains topic→partition→nsqd mappings.
• HTTP/TCP endpoints for lookup.
c. etcd cluster
• Metadata store, leader election for HA.
• Backing service for “consistence” package.
d. nsqd nodes (one per partition)
• Store messages on disk/in‑memory.
• Serve producers & consumers.
• Replication & auto‑balance logic via consistence modules.
e. nsqadmin UI
• Browser static assets + Go HTTP server.
• Reads management APIs from lookupd/nsqd.
f. Coordination library (“consistence”)
• gRPC server/client between nsqd & lookupd for cluster-wide coordination.
g. Bench & Jepsen (optional inset)
• Benchmark clients and distributed correctness tests.
Relationships and interactions
• Producers → nsqd (TCP publish)
• Consumers ← nsqd (TCP subscribe / HTTP long‑poll)
• Producers/Consumers → nsqlookupd (HTTP lookup for nsqd addresses)
• nsqd ↔ etcd (via consistence library) for metadata, leader election, replication tasks
• nsqlookupd ↔ etcd for topic registry updates
• nsqadmin ↔ nsqlookupd & nsqd (HTTP) for UI data
• Coordination between nsqd nodes via gRPC endpoints (for replication, auto‑balance)
Architectural patterns & design principles
• Decentralized topology: no single point of failure; multiple nsqd and lookupd nodes.
• Plug‑in engines: nsqd supports multiple storage engines (e.g., pebble).
• Service discovery: lookupd cluster provides dynamic routing info.
• Coordination via etcd: externalizing consensus and metadata.
• Separation of concerns: distinct binaries for queueing, discovery, UI, tooling.
• Protocol‑agnostic messaging: TCP‑based NSQ protocol; HTTP admin APIs.
1. Lay out the etcd cluster at the top (3+ nodes).
a. nsqlookupd cluster (HTTP/TCP nodes)
b. nsqd cluster (show each partition as its own node).
3. nsqadmin on the side, showing HTTP links into lookupd & nsqd.
6. Finally, annotate each component with key Go packages or paths (e.g., nsqd/backend_queue.go, consistence/coord_grpc.proto).