How hsldevcom/transitdata-partial-apc-expander-combiner works
Project type and purpose
• This is a streaming-data microservice written in TypeScript/Node.js.
• It consumes two Pulsar topics (HFP messages and partial APC messages), merges and enriches them using service-journey metadata, and publishes full APC messages to another Pulsar topic.
• It optionally reads vehicle capacity data from a relational database (for local testing via Docker Compose).
Main components of the system
a. External streaming platform (Apache Pulsar)
– Topic A: HFP consumer topic
– Topic B: partial APC consumer topic
– Topic C: full APC producer topic
b. This service’s container
– Pulsar client layer (src/pulsar.ts)
– Message consumers: one for HFP, one for partial APC
– Business-logic pipeline (src/businessLogic):
• Message parsing
• VehicleActor state machine (manages per-vehicle buffers, timers)
• Partial APC expansion (merge passenger counts)
• Service-journey metadata enrichment (uses HFP events)
• Aggregation by stop to produce full APC messages
– Database access layer (src/db):
• Reads vehicle capacities (local DB via Docker Compose or Docker secret)
– Utility modules (src/util): timers, health check, graceful exit, protobuf utilities
– Pulsar producer (full APC messages)
– HTTP health-check endpoint (default port 8080)
c. Local development dependencies (via Docker Compose)
– Mosquitto MQTT broker (for upstream testing, not used at runtime in this service)
– Vehicle-DB container (initializes capacity table from CSV via init.sql)
– Scripts to set Pulsar manager user, etc.
Architectural patterns and design principles
• Asynchronous event-driven pipeline
• Stateful “Actor” per vehicle for buffering and timers
• Domain-driven separation of concerns: parsing, processing, aggregation, persistence
• Configuration via environment variables and Docker secrets
• Test layering: unit, property, integration tests with mocked timers and Testcontainers
Detailed elements to capture
– Pulsar client instantiation (pulsar.ts)
– Two consumers and one producer configuration (environment variables)
– VehicleActor state machine managing partial APC and HFP merging
– Forced-ack timers per vehicle for stale messages
– Database lookup path (db.ts → generateVehicleCapabilities)
– Health-check route (util/healthCheck)
– Utility modules: error handling, graceful shutdown, deepEqual, protobuf serialization