# hsldevcom/transitdata-partial-apc-expander-combiner architecture

> Architecture diagram of the GitHub repository hsldevcom/transitdata-partial-apc-expander-combiner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hsldevcom/transitdata-partial-apc-expander-combiner
- Repository: https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner

## Overview

##### Project type and purpose
- This is a streaming-data microservice written in TypeScript/Node.js.
- It consumes two Pulsar topics (HFP messages and partial APC messages), merges and enriches them using service-journey metadata, and publishes full APC messages to another Pulsar topic.
- It optionally reads vehicle capacity data from a relational database (for local testing via Docker Compose).

##### Main components of the system
a. External streaming platform (Apache Pulsar)
– Topic A: HFP consumer topic
– Topic B: partial APC consumer topic
– Topic C: full APC producer topic
b. This service’s container
– Pulsar client layer (src/pulsar.ts)
– Message consumers: one for HFP, one for partial APC
– Business-logic pipeline (src/businessLogic):
- Message parsing
- VehicleActor state machine (manages per-vehicle buffers, timers)
- Partial APC expansion (merge passenger counts)
- Service-journey metadata enrichment (uses HFP events)
- Aggregation by stop to produce full APC messages
– Database access layer (src/db):
- Reads vehicle capacities (local DB via Docker Compose or Docker secret)
– Utility modules (src/util): timers, health check, graceful exit, protobuf utilities
– Pulsar producer (full APC messages)
– HTTP health-check endpoint (default port 8080)
c. Local development dependencies (via Docker Compose)
– Mosquitto MQTT broker (for upstream testing, not used at runtime in this service)
– Vehicle-DB container (initializes capacity table from CSV via init.sql)
– Scripts to set Pulsar manager user, etc.

##### Architectural patterns and design principles
- Asynchronous event-driven pipeline
- Stateful “Actor” per vehicle for buffering and timers
- Domain-driven separation of concerns: parsing, processing, aggregation, persistence
- Configuration via environment variables and Docker secrets
- Test layering: unit, property, integration tests with mocked timers and Testcontainers

##### Detailed elements to capture
– Pulsar client instantiation (pulsar.ts)
– Two consumers and one producer configuration (environment variables)
– VehicleActor state machine managing partial APC and HFP merging
– Forced-ack timers per vehicle for stale messages
– Database lookup path (db.ts → generateVehicleCapabilities)
– Health-check route (util/healthCheck)
– Utility modules: error handling, graceful shutdown, deepEqual, protobuf serialization

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    subgraph "External Systems"
        direction TB
        PulsarCluster["Apache Pulsar Cluster"]:::external
        subgraph "Pulsar Topics"
            direction TB
            HFP_TOPIC["HFP_TOPIC"]:::external
            PARTIAL_APC_TOPIC["PARTIAL_APC_TOPIC"]:::external
            FULL_APC_TOPIC["FULL_APC_TOPIC"]:::external
        end
        RelationalDB["Vehicle Capacities DB"]:::datastore
    end

    %% Service Container Layers
    subgraph "transitdata-partial-apc-expander-combiner Service" 
        direction TB

        %% Configuration and Bootstrapping
        subgraph "Service Initialization" 
            direction TB
            ConfigLoader["Configuration Loader\n(src/config.ts)"]:::serviceIO
            ServiceEntry["Service Entry Point\n(src/index.ts)"]:::serviceIO
            MainAppInit["Main App Init\n(src/main.ts)"]:::serviceIO
        end

        %% I/O Layer
        subgraph "I/O Layer" 
            direction TB
            PulsarClient["Pulsar Client Layer\n(src/pulsar.ts)"]:::serviceIO
            HFPConsumer["HFP Consumer"]:::serviceIO
            PartialApcConsumer["Partial APC Consumer"]:::serviceIO
            FullApcProducer["Full APC Producer"]:::serviceIO
            HealthCheck["HTTP Health-Check\n(src/util/healthCheck.ts)"]:::serviceIO
        end

        %% Business Logic Layer
        subgraph "Business Logic Layer"
            direction TB
            MessageParsing["Message Parsing\n(src/businessLogic/messageParsing.ts)"]:::business
            HfpHandling["HFP Message Handler\n(src/businessLogic/hfpHandling.ts)"]:::business
            PartialApcSumming["Partial APC Summation\n(src/businessLogic/partialApcSumming.ts)"]:::business
            ApcHandling["APC Aggregation Handler\n(src/businessLogic/apcHandling.ts)"]:::business
            ProducerMsgForm["Producer Message Forming\n(src/businessLogic/producerMessageForming.ts)"]:::business
            VehicleActorSM["VehicleActor State Machine\n(src/businessLogic/vehicleActor.ts)"]:::business
            VehicleProcessing["Vehicle Processing Orchestration\n(src/businessLogic/vehicleProcessing.ts)"]:::business
            MessageProcessing["Overall Pipeline\n(src/businessLogic/messageProcessing.ts)"]:::business
        end

        %% Storage Layer
        subgraph "Database Access Layer"
            direction TB
            DbAccess["DB Access Layer\n(src/db/db.ts)"]:::business
            GenVehicleCap["Generate Vehicle Capabilities\n(src/db/generateVehicleCapabilities.ts)"]:::business
        end

        %% Utility Modules
        subgraph "Utility Modules"
            direction TB
            GracefulExit["Graceful Shutdown\n(src/util/gracefulExit.ts)"]:::util
            ProtobufUtil["Protobuf Utilities\n(src/util/protobufUtil.ts)"]:::util
            ErrorUtil["Error Handling\n(src/util/error.ts)"]:::util
            SleepUtil["Timer/Sleep Utility\n(src/util/sleep.ts)"]:::util
            SplitArrayUtil["Array Split Utility\n(src/util/splitArray.ts)"]:::util
            DeepEqualUtil["Deep-Equality Utility\n(src/util/deepEqual.ts)"]:::util
        end
    end

    %% Docker Compose Orchestration
    subgraph "Local Dev Orchestration (Docker Compose)" 
        direction TB
        ComposeFile["Compose File\n(docker-compose/docker-compose-this-service.yml)"]:::orchestration
        VehicleDbInit["Vehicle-DB Init Script\n(docker-compose/config/vehicle-db/init.sql)"]:::orchestration
    end

    %% Connections
    %% Pulsar to Consumers
    PulsarCluster -->|Consumes HFP| HFPConsumer
    PulsarCluster -->|Consumes Partial APC| PartialApcConsumer
    HFP_TOPIC --> HFPConsumer
    PARTIAL_APC_TOPIC --> PartialApcConsumer

    %% Consumers to Business Logic
    HFPConsumer -->|parse HFP| MessageParsing
    PartialApcConsumer -->|parse Partial APC| MessageParsing
    MessageParsing --> HfpHandling
    MessageParsing --> PartialApcSumming

    %% Business Logic Flow
    HfpHandling --> VehicleActorSM
    PartialApcSumming --> VehicleActorSM
    VehicleActorSM --> VehicleProcessing
    VehicleProcessing --> ApcHandling
    ApcHandling --> ProducerMsgForm
    ProducerMsgForm --> MessageProcessing
    MessageProcessing --> FullApcProducer

    %% Business Logic <-> DB
    MessageProcessing -->|lookup capacities| DbAccess
    DbAccess --> GenVehicleCap
    GenVehicleCap -->|result| MessageProcessing

    %% Producer to Pulsar
    FullApcProducer -->|Publishes Full APC| FULL_APC_TOPIC
    FULL_APC_TOPIC --> PulsarCluster

    %% Health Check
    subgraph "External Monitor"
        direction TB
        Monitor["External Monitor"]:::external
    end
    Monitor -->|HTTP GET /health| HealthCheck

    %% Docker Compose links
    ComposeFile --> PulsarCluster
    ComposeFile --> RelationalDB
    ComposeFile -->|init| VehicleDbInit

    %% Click Events
    click PulsarClient "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/pulsar.ts"
    click HFPConsumer "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/pulsar.ts"
    click PartialApcConsumer "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/pulsar.ts"
    click FullApcProducer "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/pulsar.ts"
    click ServiceEntry "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/index.ts"
    click MainAppInit "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/main.ts"
    click ConfigLoader "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/config.ts"
    click MessageParsing "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/messageParsing.ts"
    click HfpHandling "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/hfpHandling.ts"
    click PartialApcSumming "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/partialApcSumming.ts"
    click ApcHandling "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/apcHandling.ts"
    click ProducerMsgForm "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/producerMessageForming.ts"
    click VehicleActorSM "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/vehicleActor.ts"
    click VehicleProcessing "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/vehicleProcessing.ts"
    click MessageProcessing "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/businessLogic/messageProcessing.ts"
    click DbAccess "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/db/db.ts"
    click GenVehicleCap "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/db/generateVehicleCapabilities.ts"
    click HealthCheck "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/healthCheck.ts"
    click GracefulExit "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/gracefulExit.ts"
    click ProtobufUtil "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/protobufUtil.ts"
    click ErrorUtil "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/error.ts"
    click SleepUtil "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/sleep.ts"
    click SplitArrayUtil "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/splitArray.ts"
    click DeepEqualUtil "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/src/util/deepEqual.ts"
    click ComposeFile "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/docker-compose/docker-compose-this-service.yml"
    click VehicleDbInit "https://github.com/hsldevcom/transitdata-partial-apc-expander-combiner/blob/main/docker-compose/config/vehicle-db/init.sql"

    %% Styles
    classDef external fill:#dddddd,stroke:#333,stroke-width:1px
    classDef serviceIO fill:#bbdefb,stroke:#0d47a1,stroke-width:1px
    classDef business fill:#c5cae9,stroke:#283593,stroke-width:1px
    classDef util fill:#dcedc8,stroke:#33691e,stroke-width:1px
    classDef datastore fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    classDef orchestration fill:#ffe0b2,stroke:#e65100,stroke-width:1px
```
