How engagetech/service-integrations works
Project’s Purpose and Type
• This is a Node.js service that bridges two external APIs—Engage and Bullhorn—by polling, webhook handling, mapping data, and persisting state in DynamoDB.
• It is a backend integration microservice (no UI).
File Structure and Technical Stack
• Top-level scripts & config
– server.js: main HTTP server entrypoint
– config.js.example, integration.json.example: runtime and mapping configuration
– gulpfile.js: build tasks (lint, tests)
– .buildkite/: CI/CD pipeline definitions and Docker scripts
• Modules by folder
– api/: wrappers for Engage and Bullhorn REST APIs plus a data-mapper
– common/: shared utilities and worker orchestrator logic
– pollers/: scheduled jobs that fetch changes from Bullhorn (placements, vacancies, etc.)
– webhooks/: HTTP handlers receiving Bullhorn event callbacks
– datastore/: DynamoDB access layer (setup, read/write to Integrations & EntityUpdates tables)
• Unit tests in unit/ mirror api/mapper behavior
README for Architecture Hints
• Two main data flows:
a) Pollers push updates into DynamoDB → common workers consume and sync to the other API
b) Webhooks receive event callbacks → enqueue or directly process updates
• Relies on local or AWS‐hosted DynamoDB with two tables (Integrations, EntityUpdates)
• External systems: Bullhorn API, Engage API, AWS (DynamoDB), Buildkite (CI/CD)
Main Components
a) External Systems
– Bullhorn API (REST + Event Subscriptions)
– Engage Public API
– AWS DynamoDB (Integrations table, EntityUpdates table)
– Buildkite (optional, for CI/CD pipeline)
b) Service Components
– server.js (HTTP listener)
– Webhook Handlers (webhooks/main.js, webhooks/vacancies.js, webhooks/workers.js)
– Pollers (pollers/main.js orchestrates, pollers/placements.js, pollers/vacancies.js, pollers/bullhorn.js)
– Common Workers (common/workers.js)
– API Layer (api/bullhorn.js, api/engage.js)
– Data Mapper (api/mapper.js)
– Datastore Layer (datastore/main.js, datastore/dynamo.js)
c) Configuration
– config.js, integration.json (holds credentials, table names, entity mapping)
Interactions and Data Flows
• Poller Flow:
1. Scheduler (cron via nodemon or PM2) invokes pollers/main.js
2. pollers/<entity>.js call Bullhorn API wrappers
3. Raw records passed through api/mapper.js to transform format
4. datastore layer writes resulting EntityUpdates to DynamoDB
5. common/workers.js reads pending updates, pushes them to Engage API or back to Bullhorn, then marks them processed
• Webhook Flow:
1. Bullhorn event subscription → HTTP POST to server.js endpoints
2. webhooks/<entity>.js parse payload and enqueue an update to DynamoDB (via datastore layer) or directly invoke common/workers
3. Further processing as in Poller Flow
• Configuration & Startup:
– server.js bootstraps express routes, loads config, connects to DynamoDB
– Uses nodemon + bunyan for local development logging
– .buildkite/pipeline.yml triggers linting (eslint), tests (unit/), Docker image build (.buildkite/scripts/Dockerfile), deploy script (.buildkite/scripts/run.sh)
Architectural Patterns & Principles
• Modular layering: API wrappers ←→ business logic (workers) ←→ persistence
• Event-driven: mix of polyglot polling and webhook callbacks
• Single responsibility: one module per concern (polling vs mapping vs persistence)
• Asynchronous queuing: DynamoDB as a transient queue for inter-service hand-off
Essential Elements Checklist
– Component labels (Pollers, Webhooks, Mapper, Workers, DynamoDB)
– Annotations for key config files (config.js, integration.json) and tables