How aaliasghar/lambda-snowflake-ingestion works
This is a prototype for event-driven CSV ingestion from S3 to Snowflake, with a separate legacy local-file-to-SQL-Server helper. The main workflow is S3 event → Lambda download and CSV parsing → normalization → Snowflake bulk load → response. The README also documents structured logging, Secrets Manager credentials, and Snowflake table setup. The handler performs its own event extraction; although src/validators.py exists, the sampled handler does not call it. The graph omits undocumented post-load processing and uncertain deployment wiring.
Main entry points: lambda_handler.py Lambda Handler · snowflake_connector.py Snowflake Loader · ingest_to_sql_server.py Local File ETL · setup_snowflake.sql Snowflake Table DDL
Components 14
S3 Ingestion
- Lambda Handler
src/lambda_handler.py - CSV Parsing
src/lambda_handler.py - Normalization
src/transformations.py
Snowflake Loading
- Snowflake Loader
src/snowflake_connector.py - Snowflake Table DDL
sql/setup_snowflake.sql - Bronze DDL
scripts/bronze/ddl_bronze.sql
Runtime Support
- Structured Logging
src/logging_config.py
Legacy Local ETL
- Local File ETL
src/ingest_to_sql_server.py
Other components
- File Uploader
- S3 Bucket
- Secrets Manager
- Snowflake Warehouse
- Local Operator
- SQL Server
Connections 13
- File Uploader → S3 Bucket: uploads CSV
- S3 Bucket → Lambda Handler: triggers invocation
- Lambda Handler → S3 Bucket: downloads object
- Lambda Handler → CSV Parsing: parses CSV
- Lambda Handler → Normalization: normalizes data
- Lambda Handler → Snowflake Loader: loads dataframe
- Snowflake Loader → Secrets Manager: fetches credentials
- Snowflake Loader → Snowflake Warehouse: bulk loads
- Lambda Handler → Structured Logging: logs events
- Snowflake Table DDL → Snowflake Warehouse: defines tables
- Bronze DDL → SQL Server: defines bronze tables
- Local Operator → Local File ETL: runs ETL
- Local File ETL → SQL Server: inserts rows