How littlek0i/snowkill works
Main Components/Modules
• Core Engine: Located in snowkill/engine.py, this module orchestrates the flow by invoking the conditions, managing notifications, and handling query terminations.
• Conditions: Modules under snowkill/condition (e.g. blocked_duration.py, cartesian_join_explosion.py, etc.) that encapsulate each specific rule used to evaluate query problems.
• Formatters: Found under snowkill/formatter (e.g. markdown.py, slack.py), these modules format notifications for various output channels.
• Storage: Modules under snowkill/storage (like abc_storage.py, postgres_table.py, snowflake_table.py, snowflake_hybrid_table.py) that help record events, prevent duplicates, and save execution state.
• Data Models & Utilities: Common components (snowkill/struct.py, error.py) for shared data abstractions and error handling.
• Examples & Tests: The example folder provides usage examples, and the test folder (with their SQL mocks and test scripts) validates the various components.
Relationships & Interaction Flow
• Query Acquisition: The process starts with gathering the list of running, queued, or blocked queries. This isn’t explicitly shown as a separate module but is part of how the engine gathers data (using internal REST calls to access Query Profiles and active locks).
• Data Processing: The core engine passes query data to each of the registered condition modules. Each condition evaluates its own criteria (like execute duration, storage spilling, etc.) and returns a verdict.
• Action & Notification: If any condition flags a query as problematic, the engine then uses one or more formatters to prepare notifications. These notifications are sent out via external services (Slack, email, etc.) and passed along to the storage modules to record the incident and avoid duplicates.
• Persistence: The storage modules (e.g., using PostgreSQL and Snowflake tables) are used to track the state of notifications and monitor if queries have already been flagged, ensuring the system doesn’t repeatedly act on the same query.
Architectural Patterns & Design Principles
• Modularity & Plug-In Architecture: The design is modular with clearly separated responsibilities for Data Gathering, Condition Evaluation, Notification Formatting, and Persistence. This makes it easy to add new conditions (e.g., “Union without ALL”) or new output formats.
• Configurability & Extensibility: The engine dynamically loads conditions, and the system supports multiple storage backends and formatter options, following principles like dependency injection and abstraction via base classes (as seen in abc_condition.py, abc_formatter.py, and abc_storage.py).
• Event-Driven Flow: Even though data flows in a fairly linear order (collect, evaluate, notify, persist), it can be thought of as event-driven where each module “reacts” to new query data.
Summary
• Flow into the “Core Engine” circle/rectangle that coordinates the process.
• Branches out to multiple “Condition Evaluators” that assess each query.
• If conditions are met, the engine accesses “Notification Formatters” to prepare messages and forwards them to “External Notification Channels.”
• Simultaneously, it interacts with “Storage” to log notifications and states.
• Enclose test and example modules in a separate note (to indicate automated testing and usage examples for clarity).