How a-vijai/cryptocurrency_project works
Project type and purpose
• This is a serverless ETL pipeline for collecting, transforming, and loading cryptocurrency data.
• It leverages AWS Step Functions to orchestrate three AWS Lambda functions and uses EventBridge to schedule daily runs.
• Data lands in Amazon S3 (raw & transformed), is loaded into Amazon Redshift Serverless, and surfaced via Looker Studio.
File structure
• Top‐level files:
• aws_lambda_extract.py – Lambda handler to call the crypto API and write raw JSON to S3
• aws_lambda_transform.py – Lambda handler to read raw data, normalize/clean, write transformed data back to S3
• aws_lambda_load.py – Lambda handler to load transformed data from S3 into Redshift Serverless
• state_machine.json – Defines the AWS Step Functions state machine with three sequential tasks
• README.md – Overview of architecture, scheduling, and BI integration
• No separate “infra” folder: state_machine.json is the only orchestration definition; Lambdas are defined inline or via an external deployment tool (not shown).
README
• Confirms:
• AWS Step Functions for ETL orchestration
• EventBridge cron trigger for daily execution
• S3 for raw and transformed storage
• Amazon Redshift Serverless for warehousing
• Looker Studio for reporting/visualization
• Clear labels on every component
1. EventBridge → Step Functions
2. Step Functions → Extract → S3 raw
3. Step Functions → Transform → S3 transformed
4. Step Functions → Load → Redshift
5. Looker → Redshift