# uzankhan/hcm-real-time-data-platform architecture

> Architecture diagram of the GitHub repository uzankhan/hcm-real-time-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uzankhan/hcm-real-time-data-platform
- Repository: https://github.com/uzankhan/hcm-real-time-data-platform

## Overview

The platform ingests jobs, labor-market benchmarks, and exchange rates into timestamped S3 objects. Its event-driven path routes S3 notifications through SNS and a Lambda router to source-filtered queues; benchmark and FX processors consume their queues. Separately, the Glue job reads the latest jobs object and appends rows to Snowflake. SQL Server is the FX destination. Edges follow explicit README architecture or shown calls; AWS provisioning and the jobs-queue consumer are not implemented here.

Main entry points: [`lambdas/router_lambda/lambda_function.py`](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/router_lambda/lambda_function.py) (Notification router), [`lambdas/process_benchmarks/lambda_function.py`](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/process_benchmarks/lambda_function.py) (Benchmark processor), [`lambdas/process_fx/lambda_function.py`](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/process_fx/lambda_function.py) (FX processor), [`lambdas/lambda_jobs_ingest/lambda_function.py`](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_jobs_ingest/lambda_function.py) (Jobs ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data ingestion"]
  node_jobs_ingest["Jobs ingestion<br/>[lambda_function.py]"]
  node_benchmark_scraper["Benchmark scraper<br/>[lambda_function.py]"]
  node_fx_ingest["FX ingestion<br/>[lambda_function.py]"]
end

subgraph group_routing["Event routing"]
  node_raw_s3[("Raw S3 objects")]
  node_sns_standard["SNS Standard"]
  node_router["Notification router<br/>[lambda_function.py]"]
  node_sns_fifo["SNS FIFO"]
  node_jobs_queue["Jobs FIFO queue"]
  node_benchmark_queue["Benchmarks FIFO queue"]
  node_fx_queue["FX FIFO queue"]
end

subgraph group_processing["Data processing"]
  node_benchmark_processor["Benchmark processor<br/>[lambda_function.py]"]
  node_fx_processor["FX processor<br/>[lambda_function.py]"]
  node_processed_s3[("Processed S3 objects")]
end

subgraph group_analytics["Analytics outputs"]
  node_glue_job["Jobs-to-Snowflake Glue job"]
  node_snowflake[("Snowflake")]
  node_sql_server[("SQL Server")]
end

node_usa_jobs(("USAJOBS API"))
node_bls(("BLS page"))
node_frankfurter(("Frankfurter API"))

node_usa_jobs -->|"fetches postings"| node_jobs_ingest
node_bls -->|"scrapes occupations"| node_benchmark_scraper
node_frankfurter -->|"fetches rates"| node_fx_ingest
node_jobs_ingest -->|"writes raw jobs"| node_raw_s3
node_benchmark_scraper -->|"writes raw benchmarks"| node_raw_s3
node_fx_ingest -->|"writes raw rates"| node_raw_s3
node_raw_s3 -->|"notifies"| node_sns_standard
node_sns_standard -->|"invokes"| node_router
node_router -->|"reads metadata"| node_raw_s3
node_router -->|"publishes routed events"| node_sns_fifo
node_sns_fifo -->|"filters jobs"| node_jobs_queue
node_sns_fifo -->|"filters benchmarks"| node_benchmark_queue
node_sns_fifo -->|"filters FX"| node_fx_queue
node_benchmark_queue -->|"delivers messages"| node_benchmark_processor
node_benchmark_processor -->|"reads raw object"| node_raw_s3
node_benchmark_processor -->|"writes processed data"| node_processed_s3
node_fx_queue -->|"delivers messages"| node_fx_processor
node_fx_processor -->|"reads raw object"| node_raw_s3
node_fx_processor -->|"inserts rates"| node_sql_server
node_raw_s3 -->|"reads latest jobs"| node_glue_job
node_glue_job -->|"appends job rows"| node_snowflake

click node_jobs_ingest "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_jobs_ingest/lambda_function.py"
click node_benchmark_scraper "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_benchmarks_scraper/lambda_function.py"
click node_fx_ingest "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_fx_ingest/lambda_function.py"
click node_router "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/router_lambda/lambda_function.py"
click node_benchmark_processor "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/process_benchmarks/lambda_function.py"
click node_fx_processor "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/process_fx/lambda_function.py"
click node_glue_job "https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/glue_scripts/jobs_to_snowflake.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_jobs_ingest,node_benchmark_scraper,node_fx_ingest toneBlue
class node_raw_s3,node_sns_standard,node_router,node_sns_fifo,node_jobs_queue,node_benchmark_queue,node_fx_queue toneAmber
class node_benchmark_processor,node_fx_processor,node_processed_s3,node_usa_jobs,node_frankfurter toneMint
class node_glue_job,node_snowflake,node_sql_server toneRose
class node_bls toneIndigo
```

## Components

### Data ingestion

- [Jobs ingestion](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_jobs_ingest/lambda_function.py): `lambdas/lambda_jobs_ingest/lambda_function.py`
- [Benchmark scraper](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_benchmarks_scraper/lambda_function.py): `lambdas/lambda_benchmarks_scraper/lambda_function.py`
- [FX ingestion](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/lambda_fx_ingest/lambda_function.py): `lambdas/lambda_fx_ingest/lambda_function.py`

### Event routing

- **Raw S3 objects**
- **SNS Standard**
- [Notification router](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/router_lambda/lambda_function.py): `lambdas/router_lambda/lambda_function.py`
- **SNS FIFO**
- **Jobs FIFO queue**
- **Benchmarks FIFO queue**
- **FX FIFO queue**

### Data processing

- [Benchmark processor](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/process_benchmarks/lambda_function.py): `lambdas/process_benchmarks/lambda_function.py`
- [FX processor](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/lambdas/process_fx/lambda_function.py): `lambdas/process_fx/lambda_function.py`
- **Processed S3 objects**

### Analytics outputs

- [Jobs-to-Snowflake Glue job](https://github.com/uzankhan/hcm-real-time-data-platform/blob/main/glue_scripts/jobs_to_snowflake.py): `glue_scripts/jobs_to_snowflake.py`
- **Snowflake**
- **SQL Server**

### Other components

- **USAJOBS API**
- **BLS page**
- **Frankfurter API**

## Connections

- USAJOBS API → Jobs ingestion: fetches postings (evidence: `README.md`)
- BLS page → Benchmark scraper: scrapes occupations (evidence: `README.md`)
- Frankfurter API → FX ingestion: fetches rates (evidence: `README.md`)
- Jobs ingestion → Raw S3 objects: writes raw jobs (evidence: `lambdas/lambda_jobs_ingest/lambda_function.py`)
- Benchmark scraper → Raw S3 objects: writes raw benchmarks (evidence: `lambdas/lambda_benchmarks_scraper/lambda_function.py`)
- FX ingestion → Raw S3 objects: writes raw rates (evidence: `lambdas/lambda_fx_ingest/lambda_function.py`)
- Raw S3 objects → SNS Standard: notifies (evidence: `README.md`)
- SNS Standard → Notification router: invokes (evidence: `README.md`)
- Notification router → Raw S3 objects: reads metadata (evidence: `lambdas/router_lambda/lambda_function.py`)
- Notification router → SNS FIFO: publishes routed events (evidence: `lambdas/router_lambda/lambda_function.py`)
- SNS FIFO → Jobs FIFO queue: filters jobs (evidence: `README.md`)
- SNS FIFO → Benchmarks FIFO queue: filters benchmarks (evidence: `README.md`)
- SNS FIFO → FX FIFO queue: filters FX (evidence: `README.md`)
- Benchmarks FIFO queue → Benchmark processor: delivers messages (evidence: `README.md`)
- Benchmark processor → Raw S3 objects: reads raw object (evidence: `lambdas/process_benchmarks/lambda_function.py`)
- Benchmark processor → Processed S3 objects: writes processed data (evidence: `lambdas/process_benchmarks/lambda_function.py`)
- FX FIFO queue → FX processor: delivers messages (evidence: `README.md`)
- FX processor → Raw S3 objects: reads raw object (evidence: `lambdas/process_fx/lambda_function.py`)
- FX processor → SQL Server: inserts rates (evidence: `lambdas/process_fx/lambda_function.py`)
- Raw S3 objects → Jobs-to-Snowflake Glue job: reads latest jobs (evidence: `glue_scripts/jobs_to_snowflake.py`)
- Jobs-to-Snowflake Glue job → Snowflake: appends job rows (evidence: `glue_scripts/jobs_to_snowflake.py`)
