How satishkumarkolamudi/snowflake works
Project Type and Purpose
• KalaYatra is a full-stack, data-intensive web application.
• Frontend: Streamlit-based interactive UI
• Backend/Data Layer: Snowflake Data Cloud for storage, transformation, and geospatial processing
• Data Sources: Public datasets (e.g., data.gov.in) ingested via CSV stages
File Structure
Top-level files and directories:
• .sql scripts: define raw tables (TOURISM_STATS_RAW.sql, CULTURAL_SITES.sql, etc.) and curated views (VW_UNTOUCHED_GEMS.sql, VW_TOURISM_SEASONALITY.sql) in Snowflake
• kalayatra_app.py: Streamlit app entry point
• popularity_score.py: Python module for computing custom metrics
• requirements.txt: Python dependencies (Snowflake connector, Snowpark, Streamlit)
• secrets.toml: Snowflake credentials (in .streamlit/)
Architectural hints:
• SQL files imply an ETL/ELT pipeline within Snowflake
• Python scripts connect to Snowflake, fetch data, and feed the UI
Extract Key Insights from README
• Data ingestion: CSVs → Snowflake stage → raw tables
• Transformation: SQL scripts → curated tables and views in CURATED_DATA schema
• UI: Streamlit queries curated views, renders maps, charts, filters
• Responsible-tourism tips and popularity scores are additional overlays
Main Components
a. External Data Sources
– data.gov.in CSV exports
b. Snowflake Data Warehouse
– Stage (@KALAYATRA_DB.RAW_DATA.KALAYATRA_STAGE)
– Schemas: RAW_DATA, CURATED_DATA
– Raw tables (TOURISM_STATS_RAW, ART_FORMS, …)
– Curated views (VW_UNTOUCHED_GEMS, VW_TOURISM_SEASONALITY, …)
c. Python Processing Layer
– popularity_score.py (Snowpark or connector-based enrichment)
d. Streamlit Frontend
– kalayatra_app.py
– Renders interactive map, filters, charts, tips
e. User’s Browser
Relationships and Data Flow
4. Python Processing (popularity_score.py) reading Curated Views and writing back metrics or passing to UI (bi-directional)
Architectural Patterns and Technologies
• ELT pattern: load raw CSVs → transform in-database with SQL
• Snowflake: Data storage, compute (Snowpark), geospatial capabilities
• Streamlit: declarative Python UI framework for data apps
• Modular Python code: separate metric computations (popularity_score.py) from UI logic
• Configuration via secrets.toml (Snowflake credentials)