# tiao553/tech-challager-2-cloud architecture

> Architecture diagram of the GitHub repository tiao553/tech-challager-2-cloud, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tiao553/tech-challager-2-cloud
- Repository: https://github.com/tiao553/tech-challager-2-cloud

## Overview

##### Project type and purpose
- This is a cloud‐based data pipeline project built on AWS. Its goal is to scrape intraday trading data from the B3 (Brazilian exchange), store raw data in S3, trigger a Lambda function to launch an AWS Glue ETL job, write processed data back to S3, catalog it, and enable queries via Athena.
- It uses Infrastructure as Code (Terraform) to provision AWS resources, Python for scraping and ETL orchestration, and serverless/event‐driven patterns throughout.

##### File structure
- Top-level scripts and docs
- readme.md – architecture overview and instructions
- job/tech-challager-2-job-agg.py – standalone aggregation script
- functions/ – two AWS Lambda functions
- fn-crawler-read-values – does the Python scraping and uploads Parquet to S3 Raw bucket
- fn-trigger-glue-job – listens to S3 Raw events and invokes the Glue job
- infrastructure/aws/ – Terraform definitions
- S3 buckets (Raw & Refined), Lambda & Glue job resources, IAM policies, ECR repo, Athena/Catalog config

##### README for insights
1) Scraping → Raw bucket
2) S3 event → Lambda → Glue ETL → Refined bucket → Glue Catalog → Athena
- It spells out technologies (boto3, pandas, selenium, pydantic, Spark SQL in Glue)

##### Main components
a. Data Scraper
- A Python process (inside Lambda or local container) using Selenium/boto3 that writes partitioned Parquet to S3 Raw
b. AWS S3 Raw bucket
- Versioned, triggers on object creation
c. Lambda “Trigger Glue”
- Subscribed to Raw bucket events, invokes the Glue job via AWS SDK
d. AWS Glue ETL job
- Performs Spark SQL transformations (grouping, summarization, column rename, date calculations)
e. AWS S3 Refined bucket
- Partitioned by date and ticker, stores processed Parquet
f. AWS Glue Data Catalog
- Automatically crawls Refined bucket, creates/updates table definitions
g. Amazon Athena
- Uses Glue Catalog metadata to run SQL queries against Refined data
h. Terraform (IaC)
- Manages provisioning of S3, Lambda, Glue, IAM, ECR, Athena, backend config

##### Relationships and data flows
- Lambda → Glue StartJobRun
- Glue job → writes to S3 Refined
- Glue Crawler (or Catalog integration) → Data Catalog
- Athena → reads via Catalog from S3 Refined

##### Architectural patterns and principles
- Serverless, event‐driven pipeline
- Infrastructure as Code for repeatable deployments
- Automatic schema cataloging and query decoupling
- Thin Lambda wrappers invoking heavier ETL in Glue
- Data lake pattern: raw zone → refined zone

##### Essential elements checklist
- Grouping by function (ingest, process, serve, deploy)
- Call out Terraform as the provisioning layer

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Ingestion"
        DScraper["Lambda: Data Scraper\n(Scraping & Parquet Upload)"]:::lambda
        RawBucket["S3 Raw Bucket\n(Parquet/raw data)"]:::bucket
    end

    subgraph "Orchestration"
        TriggerLambda["Lambda: Trigger Glue\n(Listens to S3 events)"]:::lambda
        GlueJob["AWS Glue ETL Job\n(Spark SQL Transform)"]:::glue
    end

    subgraph "Serving"
        RefinedBucket["S3 Refined Bucket\n(Partitioned Parquet)"]:::bucket
        DataCatalog["Glue Data Catalog\n(Table Definitions)"]:::catalog
        Athena["Amazon Athena\n(SQL Query Engine)"]:::athena
    end

    TerraformIaC["Terraform IaC\n(Provisioning Layer)"]:::iac

    DScraper -->|"writes Parquet to"| RawBucket
    RawBucket -->|"S3:ObjectCreated"| TriggerLambda
    TriggerLambda -->|"StartJobRun API"| GlueJob
    GlueJob -->|"writes Parquet to"| RefinedBucket
    RefinedBucket -->|"Crawl/Update"| DataCatalog
    DataCatalog -->|"Metadata Query"| Athena

    TerraformIaC -.-> DScraper
    TerraformIaC -.-> RawBucket
    TerraformIaC -.-> TriggerLambda
    TerraformIaC -.-> GlueJob
    TerraformIaC -.-> RefinedBucket
    TerraformIaC -.-> DataCatalog
    TerraformIaC -.-> Athena

    classDef lambda fill:#ffe082,stroke:#e65100,color:#e65100
    classDef bucket fill:#bbdefb,stroke:#0d47a1,color:#0d47a1
    classDef glue fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20
    classDef catalog fill:#d1c4e9,stroke:#4a148c,color:#4a148c
    classDef athena fill:#f8bbd0,stroke:#880e4f,color:#880e4f
    classDef iac fill:#eeeeee,stroke:#616161,color:#424242

    click DScraper "https://github.com/tiao553/tech-challager-2-cloud/blob/main/functions/fn-crawler-read-values/handler.py"
    click RawBucket "https://github.com/tiao553/tech-challager-2-cloud/blob/main/infrastructure/aws/storage.tf"
    click TriggerLambda "https://github.com/tiao553/tech-challager-2-cloud/blob/main/functions/fn-trigger-glue-job/handler.py"
    click GlueJob "https://github.com/tiao553/tech-challager-2-cloud/blob/main/job/tech-challager-2-job-agg.py"
    click RefinedBucket "https://github.com/tiao553/tech-challager-2-cloud/blob/main/infrastructure/aws/storage.tf"
    click DataCatalog "https://github.com/tiao553/tech-challager-2-cloud/blob/main/infrastructure/aws/crawler.tf"
    click Athena "https://github.com/tiao553/tech-challager-2-cloud/blob/main/infrastructure/aws/athena.tf"
    click TerraformIaC "https://github.com/tiao553/tech-challager-2-cloud/blob/main/infrastructure/aws/backend.tf"
```
