# yamatatsu/play-firehose-s3-glue-athena architecture

> Architecture diagram of the GitHub repository yamatatsu/play-firehose-s3-glue-athena, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yamatatsu/play-firehose-s3-glue-athena
- Repository: https://github.com/yamatatsu/play-firehose-s3-glue-athena

## Overview

##### Project type and purpose
- This is an AWS-centric data-ingestion pipeline implemented with the AWS Cloud Development Kit (CDK) in TypeScript.
- Its purpose is to ingest IoT Core messages, enrich/transform them via a Lambda, stream them through Kinesis Firehose into S3 in Parquet format (Snappy-compressed), define a Glue schema/table, and query via Athena.

##### File structure
- Top-level files:
- cdk.json: CDK app entrypoint config
- package.json / pnpm-lock.yaml: dependencies (aws-cdk libraries, AWS SDK, Jest)
- jest.config.js: test runner config
- tsconfig.json: TypeScript compiler settings
- src/ directory:
- app.ts: CDK stack instantiation
- app.DataPublisher.ts: CDK construct defining an IoT-to-Firehose publisher
- app.FirehoseEnricher.ts: CDK construct defining the Firehose delivery stream + Lambda enrichment

##### README for additional insights
- It confirms the AWS services involved and the data flow.
- It specifies partitioning by date (JST) for Athena performance, use of Glue Table for schema enforcement and Athena metadata.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CDK Project Files
    subgraph "CDK Project\n(Infrastructure as Code)"
        direction TB
        CDKJson["cdk.json"]:::code
        AppTs["src/app.ts"]:::code
        DataPub["src/app.DataPublisher.ts"]:::code
        FireEnr["src/app.FirehoseEnricher.ts"]:::code
        PkgJson["package.json"]:::code
        TsConfig["tsconfig.json"]:::code
        JestConfig["jest.config.js"]:::code
        Readme["README.md"]:::code
        LockFile["pnpm-lock.yaml"]:::code
    end

    %% AWS Resources
    subgraph "AWS Infrastructure"    
        direction TB
        IoTCore["IoT Core (Rule Action)"]:::integration
        FHIn["Firehose Delivery Stream\n(Input)"]:::analytics
        Lambda["Lambda Enricher"]:::compute
        FHOut["Firehose Delivery Stream\n(Output)"]:::analytics
        S3["S3 Parquet Bucket\n(Parquet + Snappy,\nPartition: JST date)"]:::storage
        Glue["Glue Database & Table"]:::storage
        Athena["Athena"]:::analytics
    end

    %% Data Flow
    IoTCore -->|"Rule Action"| FHIn
    FHIn -->|"Invoke enrichment"| Lambda
    Lambda -->|"Return enriched data"| FHOut
    FHOut -->|"Store as Parquet+Snappy"| S3
    Glue -->|"Points to S3 prefix"| S3
    Athena -->|"Queries via Glue Table"| Glue

    %% CDK Deployment Flow
    AppTs -->|"cdk deploy"| FHIn

    %% Click Events for code files
    click CDKJson "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/cdk.json"
    click AppTs "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/src/app.ts"
    click DataPub "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/src/app.DataPublisher.ts"
    click FireEnr "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/src/app.FirehoseEnricher.ts"
    click PkgJson "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/package.json"
    click TsConfig "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/tsconfig.json"
    click JestConfig "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/jest.config.js"
    click Readme "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/README.md"
    click LockFile "https://github.com/yamatatsu/play-firehose-s3-glue-athena/blob/main/pnpm-lock.yaml"

    %% Styles
    classDef code fill:#f9f,stroke:#333,stroke-width:1px;
    classDef integration fill:#ffeb3b,stroke:#333,stroke-width:1px;
    classDef analytics fill:#8bc34a,stroke:#333,stroke-width:1px;
    classDef compute fill:#03a9f4,stroke:#333,stroke-width:1px;
    classDef storage fill:#ff9800,stroke:#333,stroke-width:1px;
```
