How mamanullah7/listenbrainz-server works
Project type and purpose
• This is a full-stack, microservices-style web platform (“ListenBrainz Server”) for tracking and analyzing music-listening data.
• It consists of:
– A React/TypeScript Single-Page Application (frontend)
– Multiple Python WSGI/worker services (backend)
– Spark batch pipelines for heavy data processing
– A Timescale/PostgreSQL database (and auxiliary stores)
– Messaging infrastructure for real-time updates and background tasks
Main components
A. Client & UI Layer
– Web browser
– React/TypeScript SPA served as static assets via Nginx
B. Edge & Load Balancing
– Nginx (TLS termination, static file hosting, routing to API)
– Consul (service discovery) + consul-template for dynamic config
C. API Layer
– uWSGI workers hosting the Python webserver (listenbrainz/webserver + labs_api)
– RESTful endpoints, WebSockets endpoint for real-time dispatch
D. Microservices & Workers
– Metadata-cache services (Spotify, SoundCloud, Apple, Last.fm)
– Importers (lastfm_importer, spotify_reader, mbid_mapping_writer)
– Background tasks service (cron, RabbitMQ consumers)
– Timescale-writer service (writes Spark output to TimescaleDB)
– TROi and other domain-specific jobs
E. Batch-Processing Cluster
– Spark cluster (docker-spark image)
– HDFS or object storage for dumps
– Spark jobs: listens processing, incremental stats, recommendations, missing-mb data, fresh-releases, year-in-music, similarity models
F. Data Stores
– Primary PostgreSQL + Timescale extension (user/listen/stat tables)
– CouchDB (legacy/compat dumps)
– Redis (optional caching, rate limiting)
– RabbitMQ (pub/sub of listen events, task queue)
G. External Services
– MusicBrainz API
– Spotify, SoundCloud, Apple Music APIs
– Last.fm API
Relationships & interactions
– Browser → Nginx: fetch static UI; WebSocket handshake
– Browser → Nginx → uWSGI API: REST calls for data & auth
– API → PostgreSQL/Timescale: read/write user/listen data
– API → RabbitMQ: publish events (e.g., new listen)
– WebSocket worker → RabbitMQ → Browser: real-time listen dispatch
– Cron jobs → RabbitMQ or direct DB migrations (admin/sql scripts)
– Metadata-cache services → external APIs → PostgreSQL
– Spark cluster jobs consume dumps from HDFS, produce metrics → write to TimescaleDB via timescale_writer
– MBID-mapping service → reads open data dumps, writes mapping tables in DB
Architectural patterns & principles
– Layered separation: Presentation (UI), Application (API), Data (DB), Batch (Spark), Integration (external services)
– Microservices: independently deployable workers for caching, import, mapping, stats
– Event-driven: RabbitMQ broker mediates real-time updates and background processing
– Batch vs. online: Spark for large-scale analytics; API for low-latency user queries
– Infrastructure as code: Dockerfiles, docker-compose, Consul templates, GitHub Actions CI/CD
Detailed breakdown for the principal
– From Nginx, branch:
• Static → React SPA
• /api → uWSGI API container → Python webserver module
• /ws → WebSocket container → RabbitMQ listener
– At the bottom, place data stores: TimescaleDB, CouchDB, Redis. Show connections from API, services, Spark writer.