# aaliasghar/lambda-snowflake-ingestion architecture

> Architecture diagram of the GitHub repository aaliasghar/lambda-snowflake-ingestion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaliasghar/lambda-snowflake-ingestion
- Repository: https://github.com/aaliasghar/lambda-snowflake-ingestion

## Overview

This is a prototype for event-driven CSV ingestion from S3 to Snowflake, with a separate legacy local-file-to-SQL-Server helper. The main workflow is S3 event → Lambda download and CSV parsing → normalization → Snowflake bulk load → response. The README also documents structured logging, Secrets Manager credentials, and Snowflake table setup. The handler performs its own event extraction; although `src/validators.py` exists, the sampled handler does not call it. The graph omits undocumented post-load processing and uncertain deployment wiring.

Main entry points: [`src/lambda_handler.py`](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/lambda_handler.py) (Lambda Handler), [`src/snowflake_connector.py`](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/snowflake_connector.py) (Snowflake Loader), [`src/ingest_to_sql_server.py`](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/ingest_to_sql_server.py) (Local File ETL), [`sql/setup_snowflake.sql`](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/sql/setup_snowflake.sql) (Snowflake Table DDL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["S3 Ingestion"]
  node_lambda["Lambda Handler<br/>[lambda_handler.py]"]
  node_csv["CSV Parsing<br/>[lambda_handler.py]"]
  node_transform["Normalization<br/>[transformations.py]"]
end

subgraph group_snowflake["Snowflake Loading"]
  node_loader["Snowflake Loader"]
  node_tableddl["Snowflake Table DDL"]
  node_bronzedll["Bronze DDL<br/>[ddl_bronze.sql]"]
end

subgraph group_support["Runtime Support"]
  node_logging["Structured Logging<br/>[logging_config.py]"]
end

subgraph group_legacy["Legacy Local ETL"]
  node_localfiles["Local File ETL"]
end

node_uploader(("File Uploader"))
node_s3[("S3 Bucket")]
node_secrets[("Secrets Manager")]
node_warehouse[("Snowflake Warehouse")]
node_localuser(("Local Operator"))
node_sqlserver[("SQL Server")]

node_uploader -->|"uploads CSV"| node_s3
node_s3 -->|"triggers invocation"| node_lambda
node_lambda -->|"downloads object"| node_s3
node_lambda -->|"parses CSV"| node_csv
node_lambda -->|"normalizes data"| node_transform
node_lambda -->|"loads dataframe"| node_loader
node_loader -.->|"fetches credentials"| node_secrets
node_loader -->|"bulk loads"| node_warehouse
node_lambda -->|"logs events"| node_logging
node_tableddl -->|"defines tables"| node_warehouse
node_bronzedll -.->|"defines bronze tables"| node_sqlserver
node_localuser -->|"runs ETL"| node_localfiles
node_localfiles -->|"inserts rows"| node_sqlserver

click node_lambda "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/lambda_handler.py"
click node_csv "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/lambda_handler.py"
click node_transform "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/transformations.py"
click node_loader "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/snowflake_connector.py"
click node_logging "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/logging_config.py"
click node_tableddl "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/sql/setup_snowflake.sql"
click node_bronzedll "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/scripts/bronze/ddl_bronze.sql"
click node_localfiles "https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/ingest_to_sql_server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_lambda,node_csv,node_transform toneBlue
class node_loader,node_tableddl,node_bronzedll,node_s3,node_secrets,node_warehouse,node_sqlserver toneAmber
class node_logging toneMint
class node_localfiles toneRose
class node_uploader,node_localuser toneIndigo
```

## Components

### S3 Ingestion

- [Lambda Handler](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/lambda_handler.py): `src/lambda_handler.py`
- [CSV Parsing](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/lambda_handler.py): `src/lambda_handler.py`
- [Normalization](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/transformations.py): `src/transformations.py`

### Snowflake Loading

- [Snowflake Loader](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/snowflake_connector.py): `src/snowflake_connector.py`
- [Snowflake Table DDL](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/sql/setup_snowflake.sql): `sql/setup_snowflake.sql`
- [Bronze DDL](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/scripts/bronze/ddl_bronze.sql): `scripts/bronze/ddl_bronze.sql`

### Runtime Support

- [Structured Logging](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/logging_config.py): `src/logging_config.py`

### Legacy Local ETL

- [Local File ETL](https://github.com/aaliasghar/lambda-snowflake-ingestion/blob/main/src/ingest_to_sql_server.py): `src/ingest_to_sql_server.py`

### Other components

- **File Uploader**
- **S3 Bucket**
- **Secrets Manager**
- **Snowflake Warehouse**
- **Local Operator**
- **SQL Server**

## Connections

- File Uploader → S3 Bucket: uploads CSV
- S3 Bucket → Lambda Handler: triggers invocation
- Lambda Handler → S3 Bucket: downloads object
- Lambda Handler → CSV Parsing: parses CSV
- Lambda Handler → Normalization: normalizes data
- Lambda Handler → Snowflake Loader: loads dataframe
- Snowflake Loader → Secrets Manager: fetches credentials
- Snowflake Loader → Snowflake Warehouse: bulk loads
- Lambda Handler → Structured Logging: logs events
- Snowflake Table DDL → Snowflake Warehouse: defines tables
- Bronze DDL → SQL Server: defines bronze tables
- Local Operator → Local File ETL: runs ETL
- Local File ETL → SQL Server: inserts rows