# engagetech/service-integrations architecture

> Architecture diagram of the GitHub repository engagetech/service-integrations, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/engagetech/service-integrations
- Repository: https://github.com/engagetech/service-integrations

## Overview

##### Project’s Purpose and Type
- This is a Node.js service that bridges two external APIs—Engage and Bullhorn—by polling, webhook handling, mapping data, and persisting state in DynamoDB.
- It is a backend integration microservice (no UI).

##### File Structure and Technical Stack
- Top-level scripts & config
– server.js: main HTTP server entrypoint
– config.js.example, integration.json.example: runtime and mapping configuration
– gulpfile.js: build tasks (lint, tests)
– .buildkite/: CI/CD pipeline definitions and Docker scripts
- Modules by folder
– api/: wrappers for Engage and Bullhorn REST APIs plus a data-mapper
– common/: shared utilities and worker orchestrator logic
– pollers/: scheduled jobs that fetch changes from Bullhorn (placements, vacancies, etc.)
– webhooks/: HTTP handlers receiving Bullhorn event callbacks
– datastore/: DynamoDB access layer (setup, read/write to Integrations & EntityUpdates tables)
- Unit tests in unit/ mirror api/mapper behavior

##### README for Architecture Hints
- Two main data flows:
a) Pollers push updates into DynamoDB → common workers consume and sync to the other API
b) Webhooks receive event callbacks → enqueue or directly process updates
- Relies on local or AWS‐hosted DynamoDB with two tables (Integrations, EntityUpdates)
- External systems: Bullhorn API, Engage API, AWS (DynamoDB), Buildkite (CI/CD)

##### Main Components
a) External Systems
– Bullhorn API (REST + Event Subscriptions)
– Engage Public API
– AWS DynamoDB (Integrations table, EntityUpdates table)
– Buildkite (optional, for CI/CD pipeline)
b) Service Components
– server.js (HTTP listener)
– Webhook Handlers (webhooks/main.js, webhooks/vacancies.js, webhooks/workers.js)
– Pollers (pollers/main.js orchestrates, pollers/placements.js, pollers/vacancies.js, pollers/bullhorn.js)
– Common Workers (common/workers.js)
– API Layer (api/bullhorn.js, api/engage.js)
– Data Mapper (api/mapper.js)
– Datastore Layer (datastore/main.js, datastore/dynamo.js)
c) Configuration
– config.js, integration.json (holds credentials, table names, entity mapping)

##### Interactions and Data Flows
- Poller Flow:
1. Scheduler (cron via nodemon or PM2) invokes pollers/main.js
2. pollers/<entity>.js call Bullhorn API wrappers
3. Raw records passed through api/mapper.js to transform format
4. datastore layer writes resulting EntityUpdates to DynamoDB
5. common/workers.js reads pending updates, pushes them to Engage API or back to Bullhorn, then marks them processed
- Webhook Flow:
1. Bullhorn event subscription → HTTP POST to server.js endpoints
2. webhooks/<entity>.js parse payload and enqueue an update to DynamoDB (via datastore layer) or directly invoke common/workers
3. Further processing as in Poller Flow
- Configuration & Startup:
– server.js bootstraps express routes, loads config, connects to DynamoDB
– Uses nodemon + bunyan for local development logging
– .buildkite/pipeline.yml triggers linting (eslint), tests (unit/), Docker image build (.buildkite/scripts/Dockerfile), deploy script (.buildkite/scripts/run.sh)

##### Architectural Patterns & Principles
- Modular layering: API wrappers ←→ business logic (workers) ←→ persistence
- Event-driven: mix of polyglot polling and webhook callbacks
- Single responsibility: one module per concern (polling vs mapping vs persistence)
- Asynchronous queuing: DynamoDB as a transient queue for inter-service hand-off

##### Essential Elements Checklist
– Component labels (Pollers, Webhooks, Mapper, Workers, DynamoDB)
– Annotations for key config files (config.js, integration.json) and tables

## Diagram (Mermaid)

```mermaid
flowchart TD
  subgraph "External Systems"
    BH["Bullhorn API"]:::external
    EN["Engage API"]:::external
    DB["DynamoDB\n(Integrations, EntityUpdates)"]:::persistence
    BK["Buildkite CI/CD"]:::external
  end

  subgraph "Configuration"
    CFG1["config.js.example"]:::config
    CFG2["integration.json.example"]:::config
  end

  subgraph "Service Components"
    S["server.js"]:::api
    subgraph "Webhook Receiver"
      WH1["webhooks/main.js"]:::api
      WH2["webhooks/vacancies.js"]:::api
      WH3["webhooks/workers.js"]:::api
    end
    subgraph "Poller Service"
      P0["pollers/main.js"]:::logic
      P1["pollers/bullhorn.js"]:::logic
      P2["pollers/placements.js"]:::logic
      P3["pollers/vacancies.js"]:::logic
    end
    subgraph "API Layer"
      API1["api/bullhorn.js"]:::logic
      API2["api/engage.js"]:::logic
    end
    M["api/mapper.js"]:::logic
    subgraph "Datastore Layer"
      DS1["datastore/main.js"]:::persistence
      DS2["datastore/dynamo.js"]:::persistence
    end
    W["common/workers.js"]:::logic
    G["gulpfile.js"]:::cicd
    subgraph "CI/CD Pipeline"
      CI1[".buildkite/pipeline.yml"]:::cicd
      CI2[".buildkite/scripts/Dockerfile"]:::cicd
      CI3[".buildkite/scripts/docker.sh"]:::cicd
      CI4[".buildkite/scripts/run.sh"]:::cicd
    end
  end

  BH --> S
  S --> WH1 --> DS1
  S --> WH2 --> DS1
  S --> WH3 --> DS1
  P0 --> P1 --> API1 --> M --> DS1
  P0 --> P2 --> API1
  P0 --> P3 --> API1
  DS1 --> DS2 --> DB
  DB --> W
  W --> API2 --> EN
  W --> API1 --> BH
  CFG1 --> S
  CFG2 --> S
  CFG1 --> P0
  CFG2 --> P0
  G --> CI1
  CI1 --> CI2 --> CI3 --> CI4

  click S "https://github.com/engagetech/service-integrations/blob/master/server.js"
  click WH1 "https://github.com/engagetech/service-integrations/blob/master/webhooks/main.js"
  click WH2 "https://github.com/engagetech/service-integrations/blob/master/webhooks/vacancies.js"
  click WH3 "https://github.com/engagetech/service-integrations/blob/master/webhooks/workers.js"
  click P0 "https://github.com/engagetech/service-integrations/blob/master/pollers/main.js"
  click P1 "https://github.com/engagetech/service-integrations/blob/master/pollers/bullhorn.js"
  click P2 "https://github.com/engagetech/service-integrations/blob/master/pollers/placements.js"
  click P3 "https://github.com/engagetech/service-integrations/blob/master/pollers/vacancies.js"
  click W "https://github.com/engagetech/service-integrations/blob/master/common/workers.js"
  click API1 "https://github.com/engagetech/service-integrations/blob/master/api/bullhorn.js"
  click API2 "https://github.com/engagetech/service-integrations/blob/master/api/engage.js"
  click M "https://github.com/engagetech/service-integrations/blob/master/api/mapper.js"
  click DS1 "https://github.com/engagetech/service-integrations/blob/master/datastore/main.js"
  click DS2 "https://github.com/engagetech/service-integrations/blob/master/datastore/dynamo.js"
  click CFG1 "https://github.com/engagetech/service-integrations/blob/master/config.js.example"
  click CFG2 "https://github.com/engagetech/service-integrations/blob/master/integration.json.example"
  click CI1 "https://github.com/engagetech/service-integrations/blob/master/.buildkite/pipeline.yml"
  click CI2 "https://github.com/engagetech/service-integrations/tree/master/.buildkite/scripts/Dockerfile"
  click CI3 "https://github.com/engagetech/service-integrations/blob/master/.buildkite/scripts/docker.sh"
  click CI4 "https://github.com/engagetech/service-integrations/blob/master/.buildkite/scripts/run.sh"
  click G "https://github.com/engagetech/service-integrations/blob/master/gulpfile.js"

  classDef external fill:#888888,stroke:#333,stroke-width:1px,color:#fff;
  classDef api fill:#bbdefb,stroke:#1e88e5,stroke-width:1px;
  classDef logic fill:#ffcc80,stroke:#fb8c00,stroke-width:1px;
  classDef persistence fill:#c8e6c9,stroke:#43a047,stroke-width:1px;
  classDef config fill:#ffe0b2,stroke:#ffb300,stroke-width:1px;
  classDef cicd fill:#e0e0e0,stroke:#757575,stroke-width:1px;
```
