How tiao553/tech-challager-2-cloud works
Project type and purpose
• This is a cloud‐based data pipeline project built on AWS. Its goal is to scrape intraday trading data from the B3 (Brazilian exchange), store raw data in S3, trigger a Lambda function to launch an AWS Glue ETL job, write processed data back to S3, catalog it, and enable queries via Athena.
• It uses Infrastructure as Code (Terraform) to provision AWS resources, Python for scraping and ETL orchestration, and serverless/event‐driven patterns throughout.
File structure
• Top-level scripts and docs
• readme.md – architecture overview and instructions
• job/tech-challager-2-job-agg.py – standalone aggregation script
• functions/ – two AWS Lambda functions
• fn-crawler-read-values – does the Python scraping and uploads Parquet to S3 Raw bucket
• fn-trigger-glue-job – listens to S3 Raw events and invokes the Glue job
• infrastructure/aws/ – Terraform definitions
• S3 buckets (Raw & Refined), Lambda & Glue job resources, IAM policies, ECR repo, Athena/Catalog config
README for insights
1) Scraping → Raw bucket
2) S3 event → Lambda → Glue ETL → Refined bucket → Glue Catalog → Athena
• It spells out technologies (boto3, pandas, selenium, pydantic, Spark SQL in Glue)
Main components
a. Data Scraper
• A Python process (inside Lambda or local container) using Selenium/boto3 that writes partitioned Parquet to S3 Raw
b. AWS S3 Raw bucket
• Versioned, triggers on object creation
c. Lambda “Trigger Glue”
• Subscribed to Raw bucket events, invokes the Glue job via AWS SDK
d. AWS Glue ETL job
• Performs Spark SQL transformations (grouping, summarization, column rename, date calculations)
e. AWS S3 Refined bucket
• Partitioned by date and ticker, stores processed Parquet
f. AWS Glue Data Catalog
• Automatically crawls Refined bucket, creates/updates table definitions
g. Amazon Athena
• Uses Glue Catalog metadata to run SQL queries against Refined data
h. Terraform (IaC)
• Manages provisioning of S3, Lambda, Glue, IAM, ECR, Athena, backend config
Relationships and data flows
• Lambda → Glue StartJobRun
• Glue job → writes to S3 Refined
• Glue Crawler (or Catalog integration) → Data Catalog
• Athena → reads via Catalog from S3 Refined
Architectural patterns and principles
• Serverless, event‐driven pipeline
• Infrastructure as Code for repeatable deployments
• Automatic schema cataloging and query decoupling
• Thin Lambda wrappers invoking heavier ETL in Glue
• Data lake pattern: raw zone → refined zone
Essential elements checklist
• Grouping by function (ingest, process, serve, deploy)
• Call out Terraform as the provisioning layer