How waffleliew/mrt-crowd-tracker works
Project Type and Purpose
• This is a full-stack, real-time crowd-density tracking application for MRT stations.
• Frontend: Next.js map UI that displays live crowd data.
• Backend: Spring Boot service that fetches LTA DataMall API, processes via Kafka, stores in MongoDB, and serves data to the frontend.
• Infrastructure: Kafka (with Zookeeper), MongoDB (with Mongo-Express UI), all orchestrated via Docker Compose.
Main Components
a. External Data Source
• LTA DataMall REST API
b. Backend Service (Spring Boot)
• Kafka Producer (fetches from LTA, pushes to Kafka topic)
• Kafka Consumer (reads topic, processes messages)
• MongoConfig and Mongo Repository (stores processed crowd data)
• WebSocketEventListener (pushes real-time updates to frontend via WebSocket)
• REST Controllers (optional: serves on-demand crowd data via HTTP API)
c. Messaging Layer
• Zookeeper
• Kafka Broker (topic: crowd-density-updates)
d. Data Storage
• MongoDB
• Mongo-Express (Admin UI)
e. Frontend (Next.js)
• UI component (reads from WebSocket or REST endpoint)
• Static CSV data source for station locations (library/mrt_lrt_data.csv)
f. Container Orchestration
• Docker Compose network grouping all the above services
Relationships and Data Flows
1. Spring Boot “KafkaProducerService” polls LTA DataMall → produces JSON messages to Kafka topic.
2. Spring Boot “KafkaConsumerService” subscribes to the same topic → processes and writes records to MongoDB via “CrowdDensityRepo.”
3. WebSocketEventListener emits updates downstream to any connected frontend clients.
4. Frontend connects via:
– WebSocket (real-time push)
– Or REST GET /crowddensity (initial load or fallback)
5. Mongo-Express connects to MongoDB for admin UI on port 8081.
Architectural Patterns and Design Principles
• Event-Driven Architecture (Kafka pub/sub)
• Single Page Application (Next.js)
• Containerization (Docker, Docker Compose)
• Separation of concerns:
• Data Ingestion (producer)
• Data Processing (consumer, repository)
• Data Presentation (frontend)
• Infrastructure as code (docker-compose.yaml)