# littlek0i/snowkill architecture

> Architecture diagram of the GitHub repository littlek0i/snowkill, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/littlek0i/snowkill
- Repository: https://github.com/littlek0i/snowkill

## Overview

##### Main Components/Modules
- Core Engine: Located in snowkill/engine.py, this module orchestrates the flow by invoking the conditions, managing notifications, and handling query terminations.
- Conditions: Modules under snowkill/condition (e.g. blocked_duration.py, cartesian_join_explosion.py, etc.) that encapsulate each specific rule used to evaluate query problems.
- Formatters: Found under snowkill/formatter (e.g. markdown.py, slack.py), these modules format notifications for various output channels.
- Storage: Modules under snowkill/storage (like abc_storage.py, postgres_table.py, snowflake_table.py, snowflake_hybrid_table.py) that help record events, prevent duplicates, and save execution state.
- Data Models & Utilities: Common components (snowkill/struct.py, error.py) for shared data abstractions and error handling.
- Examples & Tests: The example folder provides usage examples, and the test folder (with their SQL mocks and test scripts) validates the various components.

##### Relationships & Interaction Flow
- Query Acquisition: The process starts with gathering the list of running, queued, or blocked queries. This isn’t explicitly shown as a separate module but is part of how the engine gathers data (using internal REST calls to access Query Profiles and active locks).
- Data Processing: The core engine passes query data to each of the registered condition modules. Each condition evaluates its own criteria (like execute duration, storage spilling, etc.) and returns a verdict.
- Action & Notification: If any condition flags a query as problematic, the engine then uses one or more formatters to prepare notifications. These notifications are sent out via external services (Slack, email, etc.) and passed along to the storage modules to record the incident and avoid duplicates.
- Persistence: The storage modules (e.g., using PostgreSQL and Snowflake tables) are used to track the state of notifications and monitor if queries have already been flagged, ensuring the system doesn’t repeatedly act on the same query.

##### Architectural Patterns & Design Principles
- Modularity & Plug-In Architecture: The design is modular with clearly separated responsibilities for Data Gathering, Condition Evaluation, Notification Formatting, and Persistence. This makes it easy to add new conditions (e.g., “Union without ALL”) or new output formats.
- Configurability & Extensibility: The engine dynamically loads conditions, and the system supports multiple storage backends and formatter options, following principles like dependency injection and abstraction via base classes (as seen in abc_condition.py, abc_formatter.py, and abc_storage.py).
- Event-Driven Flow: Even though data flows in a fairly linear order (collect, evaluate, notify, persist), it can be thought of as event-driven where each module “reacts” to new query data.

##### Summary
- Flow into the “Core Engine” circle/rectangle that coordinates the process.
- Branches out to multiple “Condition Evaluators” that assess each query.
- If conditions are met, the engine accesses “Notification Formatters” to prepare messages and forwards them to “External Notification Channels.”
- Simultaneously, it interacts with “Storage” to log notifications and states.
- Enclose test and example modules in a separate note (to indicate automated testing and usage examples for clarity).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        DC1["Snowflake Data Cloud"]:::external
    end

    %% Query Collection Layer
    subgraph "Query Collection"
        QDC["Query Data Collector"]:::collector
    end

    %% Processing Layer
    subgraph "Processing"
        CE["Core Engine"]:::engine

        subgraph "Data Models & Utilities"
            DS["Data Structures"]:::dataModel
            EH["Error Handling"]:::dataModel
        end
        
        %% Condition Evaluators
        subgraph "Evaluation"
            C["Condition Evaluators"]:::conditions
            BC["Base Condition Interface"]:::conditionInterface
            BD["Blocked Duration"]:::condition
            CJ["Cartesian Join Explosion"]:::condition
            ES["Estimated Scan Duration"]:::condition
            ED["Execute Duration"]:::condition
            JE["Join Explosion"]:::condition
            QD["Queued Duration"]:::condition
            SS["Storage Spilling"]:::condition
            UA["Union Without All"]:::condition
            C --> BC
            C --> BD
            C --> CJ
            C --> ES
            C --> ED
            C --> JE
            C --> QD
            C --> SS
            C --> UA
        end

        %% Notification Formatting
        subgraph "Notifications"
            F["Notification Formatters"]:::formatters
            BF["Base Formatter Interface"]:::formatterInterface
            MF["Markdown Formatter"]:::formatter
            SF["Slack Formatter"]:::formatter
            F --> BF
            F --> MF
            F --> SF

            NTC["External Notification Channels"]:::extNotif
            F --> NTC
        end

        %% Persistence Storage
        subgraph "Storage Layer"
            S["Persistence Storage"]:::storage
            BS["Base Storage Interface"]:::storageInterface
            PS["Postgres Table Storage"]:::storageDetail
            ST["Snowflake Table Storage"]:::storageDetail
            SH["Snowflake Hybrid Table Storage"]:::storageDetail
            S --> BS
            S --> PS
            S --> ST
            S --> SH
        end

        %% Connections within Processing Layer
        CE --> C
        CE --> F
        CE --> S
        CE --> DS
        CE --> EH
    end

    %% Overall Flow Connections
    DC1 --> QDC
    QDC --> CE
    CE -->|"terminates"| DC1

    %% Others: Examples & Tests
    subgraph "Others"
        E["Usage Examples"]:::others
        T["Test Suite"]:::others
    end

    %% Click Events
    click CE "https://github.com/littlek0i/snowkill/blob/master/snowkill/engine.py"
    click BC "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/abc_condition.py"
    click BD "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/blocked_duration.py"
    click CJ "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/cartesian_join_explosion.py"
    click ES "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/estimated_scan_duration.py"
    click ED "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/execute_duration.py"
    click JE "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/join_explosion.py"
    click QD "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/queued_duration.py"
    click SS "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/storage_spilling.py"
    click UA "https://github.com/littlek0i/snowkill/blob/master/snowkill/condition/union_without_all.py"
    click BF "https://github.com/littlek0i/snowkill/blob/master/snowkill/formatter/abc_formatter.py"
    click MF "https://github.com/littlek0i/snowkill/blob/master/snowkill/formatter/markdown.py"
    click SF "https://github.com/littlek0i/snowkill/blob/master/snowkill/formatter/slack.py"
    click BS "https://github.com/littlek0i/snowkill/blob/master/snowkill/storage/abc_storage.py"
    click PS "https://github.com/littlek0i/snowkill/blob/master/snowkill/storage/postgres_table.py"
    click ST "https://github.com/littlek0i/snowkill/blob/master/snowkill/storage/snowflake_table.py"
    click SH "https://github.com/littlek0i/snowkill/blob/master/snowkill/storage/snowflake_hybrid_table.py"
    click DS "https://github.com/littlek0i/snowkill/blob/master/snowkill/struct.py"
    click EH "https://github.com/littlek0i/snowkill/blob/master/snowkill/error.py"
    click E "https://github.com/littlek0i/snowkill/tree/master/example/"
    click T "https://github.com/littlek0i/snowkill/tree/master/test/"

    %% Styles
    classDef external fill:#ffcc00,stroke:#333,stroke-width:2px
    classDef collector fill:#cce5ff,stroke:#333,stroke-width:2px
    classDef engine fill:#f6c1e1,stroke:#333,stroke-width:2px
    classDef conditions fill:#d1f0f0,stroke:#333,stroke-width:2px
    classDef conditionInterface fill:#aee9e9,stroke:#333,stroke-width:2px
    classDef condition fill:#c0eafc,stroke:#333,stroke-width:2px
    classDef formatters fill:#d0f0d0,stroke:#333,stroke-width:2px
    classDef formatterInterface fill:#a9e6a9,stroke:#333,stroke-width:2px
    classDef formatter fill:#bdf7bd,stroke:#333,stroke-width:2px
    classDef extNotif fill:#ffe79a,stroke:#333,stroke-width:2px
    classDef storage fill:#fff3b0,stroke:#333,stroke-width:2px
    classDef storageInterface fill:#ffc299,stroke:#333,stroke-width:2px
    classDef storageDetail fill:#ffe6cc,stroke:#333,stroke-width:2px
    classDef dataModel fill:#dfefff,stroke:#333,stroke-width:2px
    classDef others fill:#e6e6e6,stroke:#333,stroke-width:2px"
```
