How uzankhan/hcm-real-time-data-platform works
The platform ingests jobs, labor-market benchmarks, and exchange rates into timestamped S3 objects. Its event-driven path routes S3 notifications through SNS and a Lambda router to source-filtered queues; benchmark and FX processors consume their queues. Separately, the Glue job reads the latest jobs object and appends rows to Snowflake. SQL Server is the FX destination. Edges follow explicit README architecture or shown calls; AWS provisioning and the jobs-queue consumer are not implemented here.
Main entry points: router_lambda/lambda_function.py Notification router · process_benchmarks/lambda_function.py Benchmark processor · process_fx/lambda_function.py FX processor · lambda_jobs_ingest/lambda_function.py Jobs ingestion
Components 19
Data ingestion
- Jobs ingestion
lambdas/lambda_jobs_ingest/lambda_function.py - Benchmark scraper
lambdas/lambda_benchmarks_scraper/lambda_function.py - FX ingestion
lambdas/lambda_fx_ingest/lambda_function.py
Event routing
- Raw S3 objects
- SNS Standard
- Notification router
lambdas/router_lambda/lambda_function.py - SNS FIFO
- Jobs FIFO queue
- Benchmarks FIFO queue
- FX FIFO queue
Data processing
- Benchmark processor
lambdas/process_benchmarks/lambda_function.py - FX processor
lambdas/process_fx/lambda_function.py - Processed S3 objects
Analytics outputs
- Jobs-to-Snowflake Glue job
glue_scripts/jobs_to_snowflake.py - Snowflake
- SQL Server
Other components
- USAJOBS API
- BLS page
- Frankfurter API
Connections 21
- USAJOBS API → Jobs ingestion: fetches postings
README.md - BLS page → Benchmark scraper: scrapes occupations
README.md - Frankfurter API → FX ingestion: fetches rates
README.md - Jobs ingestion → Raw S3 objects: writes raw jobs
lambdas/lambda_jobs_ingest/lambda_function.py - Benchmark scraper → Raw S3 objects: writes raw benchmarks
lambdas/lambda_benchmarks_scraper/lambda_function.py - FX ingestion → Raw S3 objects: writes raw rates
lambdas/lambda_fx_ingest/lambda_function.py - Raw S3 objects → SNS Standard: notifies
README.md - SNS Standard → Notification router: invokes
README.md - Notification router → Raw S3 objects: reads metadata
lambdas/router_lambda/lambda_function.py - Notification router → SNS FIFO: publishes routed events
lambdas/router_lambda/lambda_function.py - SNS FIFO → Jobs FIFO queue: filters jobs
README.md - SNS FIFO → Benchmarks FIFO queue: filters benchmarks
README.md - SNS FIFO → FX FIFO queue: filters FX
README.md - Benchmarks FIFO queue → Benchmark processor: delivers messages
README.md - Benchmark processor → Raw S3 objects: reads raw object
lambdas/process_benchmarks/lambda_function.py - Benchmark processor → Processed S3 objects: writes processed data
lambdas/process_benchmarks/lambda_function.py - FX FIFO queue → FX processor: delivers messages
README.md - FX processor → Raw S3 objects: reads raw object
lambdas/process_fx/lambda_function.py - FX processor → SQL Server: inserts rates
lambdas/process_fx/lambda_function.py - Raw S3 objects → Jobs-to-Snowflake Glue job: reads latest jobs
glue_scripts/jobs_to_snowflake.py - Jobs-to-Snowflake Glue job → Snowflake: appends job rows
glue_scripts/jobs_to_snowflake.py