How yamatatsu/play-firehose-s3-glue-athena works
Project type and purpose
• This is an AWS-centric data-ingestion pipeline implemented with the AWS Cloud Development Kit (CDK) in TypeScript.
• Its purpose is to ingest IoT Core messages, enrich/transform them via a Lambda, stream them through Kinesis Firehose into S3 in Parquet format (Snappy-compressed), define a Glue schema/table, and query via Athena.
File structure
• Top-level files:
• cdk.json: CDK app entrypoint config
• package.json / pnpm-lock.yaml: dependencies (aws-cdk libraries, AWS SDK, Jest)
• jest.config.js: test runner config
• tsconfig.json: TypeScript compiler settings
• src/ directory:
• app.ts: CDK stack instantiation
• app.DataPublisher.ts: CDK construct defining an IoT-to-Firehose publisher
• app.FirehoseEnricher.ts: CDK construct defining the Firehose delivery stream + Lambda enrichment
README for additional insights
• It confirms the AWS services involved and the data flow.
• It specifies partitioning by date (JST) for Athena performance, use of Glue Table for schema enforcement and Athena metadata.