# blackuprise/crunchit architecture

> Architecture diagram of the GitHub repository blackuprise/crunchit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blackuprise/crunchit
- Repository: https://github.com/blackuprise/crunchit

## Overview

CrunchIt consumes Kafka messages through a query-specific pipeline, decodes and evaluates SQL filters/aggregations, tracks entity state, and periodically materializes aggregate views in a selected database. Queries are managed and activated through the configuration UI; a separate dashboard reports pipeline metrics. The README documents decoder choices, Redis deduplication, deletion, multiple storage adapters, and query lifecycle. Some implementation links are based on README descriptions because their source was not excerpted.

Main entry points: [`lib/crunch_it/infrastructure/kafka/pipeline.ex`](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/pipeline.ex) (Kafka pipeline), [`lib/crunch_it/query_config.ex`](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/query_config.ex) (Query lifecycle), [`lib/crunch_it/application/materialization/commands/upsert_view.ex`](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/materialization/commands/upsert_view.ex) (View updates), [`lib/crunch_it/application/sql/compiler.ex`](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/compiler.ex) (SQL compiler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_management["Query management"]
  node_query_ui["Query UI<br/>[config_live.ex]"]
  node_query_config["Query lifecycle<br/>[query_config.ex]"]
  node_query_model["Query definition<br/>[query.ex]"]
end

subgraph group_ingestion["Kafka ingestion"]
  node_pipeline["Kafka pipeline<br/>[pipeline.ex]"]
  node_decoders["Message decoders<br/>[decoder.ex]"]
  node_redis_dedup[("Redis deduplication<br/>[dedup.ex]")]
end

subgraph group_compute["Query and aggregation"]
  node_sql_parser["SQL parser<br/>[parser.ex]"]
  node_sql_compiler["SQL compiler<br/>[compiler.ex]"]
  node_expression["SQL expressions<br/>[expression.ex]"]
  node_compiled_query["Compiled query<br/>[compiled_query.ex]"]
  node_evaluator["SQL evaluator<br/>[evaluator.ex]"]
  node_aggregation["Aggregation behaviors<br/>[incremental.ex]"]
end

subgraph group_materialization["State and storage"]
  node_window_state[("Window state<br/>[ets_state.ex]")]
  node_flush["Window flush<br/>[flush_server.ex]"]
  node_view_commands["View updates<br/>[upsert_view.ex]"]
  node_postgres[("Postgres materialization<br/>[upsert.ex]")]
  node_storage_adapters["Storage adapters<br/>[adapter.ex]"]
  node_id_tracking["Associated ID tracking<br/>[store_ids.ex]"]
end

subgraph group_operations["Dashboard and operations"]
  node_dashboard["Pipeline dashboard<br/>[pipeline_live.ex]"]
  node_metrics[("Metrics store<br/>[metrics_store.ex]")]
  node_app["Application supervision<br/>[application.ex]"]
end

node_operator(("Operator"))
node_kafka["Kafka topic"]

node_operator -->|"manages queries"| node_query_ui
node_query_ui -.->|"submits changes"| node_query_config
node_query_config -->|"compiles query"| node_sql_compiler
node_query_config -.->|"starts pipeline with"| node_compiled_query
node_query_model -.->|"contains SQL"| node_sql_parser
node_sql_parser -.->|"parses query for"| node_sql_compiler
node_sql_compiler -.->|"compiles expressions"| node_expression
node_compiled_query -->|"configures"| node_pipeline
node_kafka -->|"provides messages"| node_pipeline
node_pipeline -.->|"decodes messages"| node_decoders
node_pipeline -->|"evaluates messages"| node_evaluator
node_pipeline -->|"checks and marks keys"| node_redis_dedup
node_evaluator -.->|"updates aggregates"| node_aggregation
node_pipeline -->|"reads and writes state"| node_window_state
node_pipeline -->|"requests flush"| node_flush
node_flush -.->|"materializes window"| node_view_commands
node_view_commands -.->|"upserts view"| node_postgres
node_view_commands -.->|"uses selected backend"| node_storage_adapters
node_pipeline -.->|"tracks associated IDs"| node_id_tracking
node_pipeline -->|"records metrics"| node_metrics
node_dashboard -->|"reads metrics"| node_metrics
node_dashboard -->|"fetches lag"| node_kafka
node_app -->|"starts service"| node_query_config
node_app -->|"starts service"| node_metrics

click node_query_ui "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/dashboard/config_live.ex"
click node_query_config "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/query_config.ex"
click node_query_model "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/domain/query.ex"
click node_sql_parser "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/parser.ex"
click node_sql_compiler "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/compiler.ex"
click node_expression "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/expression.ex"
click node_compiled_query "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/domain/compiled_query.ex"
click node_pipeline "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/pipeline.ex"
click node_decoders "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/decoder.ex"
click node_evaluator "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/evaluator.ex"
click node_aggregation "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/aggregation/incremental.ex"
click node_window_state "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/ets_state.ex"
click node_flush "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/flush_server.ex"
click node_view_commands "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/materialization/commands/upsert_view.ex"
click node_postgres "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/postgres/upsert.ex"
click node_storage_adapters "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/storage/adapter.ex"
click node_redis_dedup "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/redis/dedup.ex"
click node_id_tracking "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/materialization/commands/store_ids.ex"
click node_dashboard "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/dashboard/pipeline_live.ex"
click node_metrics "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/dashboard/metrics_store.ex"
click node_app "https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application.ex"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_query_ui,node_query_config,node_query_model toneBlue
class node_pipeline,node_decoders,node_redis_dedup toneAmber
class node_sql_parser,node_sql_compiler,node_expression,node_compiled_query,node_evaluator,node_aggregation toneMint
class node_window_state,node_flush,node_view_commands,node_postgres,node_storage_adapters,node_id_tracking toneRose
class node_dashboard,node_metrics,node_app,node_operator,node_kafka toneIndigo
```

## Components

### Query management

- [Query UI](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/dashboard/config_live.ex): `lib/crunch_it/dashboard/config_live.ex`
- [Query lifecycle](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/query_config.ex): `lib/crunch_it/query_config.ex`
- [Query definition](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/domain/query.ex): `lib/crunch_it/domain/query.ex`

### Kafka ingestion

- [Kafka pipeline](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/pipeline.ex): `lib/crunch_it/infrastructure/kafka/pipeline.ex`
- [Message decoders](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/decoder.ex): `lib/crunch_it/infrastructure/kafka/decoder.ex`
- [Redis deduplication](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/redis/dedup.ex): `lib/crunch_it/infrastructure/redis/dedup.ex`

### Query and aggregation

- [SQL parser](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/parser.ex): `lib/crunch_it/application/sql/parser.ex`
- [SQL compiler](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/compiler.ex): `lib/crunch_it/application/sql/compiler.ex`
- [SQL expressions](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/expression.ex): `lib/crunch_it/application/sql/expression.ex`
- [Compiled query](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/domain/compiled_query.ex): `lib/crunch_it/domain/compiled_query.ex`
- [SQL evaluator](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/sql/evaluator.ex): `lib/crunch_it/application/sql/evaluator.ex`
- [Aggregation behaviors](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/aggregation/incremental.ex): `lib/crunch_it/aggregation/incremental.ex`

### State and storage

- [Window state](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/ets_state.ex): `lib/crunch_it/infrastructure/kafka/ets_state.ex`
- [Window flush](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/kafka/flush_server.ex): `lib/crunch_it/infrastructure/kafka/flush_server.ex`
- [View updates](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/materialization/commands/upsert_view.ex): `lib/crunch_it/application/materialization/commands/upsert_view.ex`
- [Postgres materialization](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/infrastructure/postgres/upsert.ex): `lib/crunch_it/infrastructure/postgres/upsert.ex`
- [Storage adapters](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/storage/adapter.ex): `lib/crunch_it/storage/adapter.ex`
- [Associated ID tracking](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application/materialization/commands/store_ids.ex): `lib/crunch_it/application/materialization/commands/store_ids.ex`

### Dashboard and operations

- [Pipeline dashboard](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/dashboard/pipeline_live.ex): `lib/crunch_it/dashboard/pipeline_live.ex`
- [Metrics store](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/dashboard/metrics_store.ex): `lib/crunch_it/dashboard/metrics_store.ex`
- [Application supervision](https://github.com/blackuprise/crunchit/blob/main/lib/crunch_it/application.ex): `lib/crunch_it/application.ex`

### Other components

- **Operator**
- **Kafka topic**

## Connections

- Operator → Query UI: manages queries
- Query UI → Query lifecycle: submits changes
- Query lifecycle → SQL compiler: compiles query (evidence: `lib/crunch_it/query_config.ex`)
- Query lifecycle → Compiled query: starts pipeline with
- Query definition → SQL parser: contains SQL
- SQL parser → SQL compiler: parses query for
- SQL compiler → SQL expressions: compiles expressions
- Compiled query → Kafka pipeline: configures (evidence: `lib/crunch_it/infrastructure/kafka/pipeline.ex`)
- Kafka topic → Kafka pipeline: provides messages
- Kafka pipeline → Message decoders: decodes messages
- Kafka pipeline → SQL evaluator: evaluates messages (evidence: `lib/crunch_it/infrastructure/kafka/pipeline.ex`)
- Kafka pipeline → Redis deduplication: checks and marks keys (evidence: `lib/crunch_it/infrastructure/kafka/pipeline.ex`)
- SQL evaluator → Aggregation behaviors: updates aggregates
- Kafka pipeline → Window state: reads and writes state (evidence: `lib/crunch_it/infrastructure/kafka/pipeline.ex`)
- Kafka pipeline → Window flush: requests flush (evidence: `lib/crunch_it/infrastructure/kafka/pipeline.ex`)
- Window flush → View updates: materializes window
- View updates → Postgres materialization: upserts view
- View updates → Storage adapters: uses selected backend
- Kafka pipeline → Associated ID tracking: tracks associated IDs
- Kafka pipeline → Metrics store: records metrics (evidence: `lib/crunch_it/infrastructure/kafka/pipeline.ex`)
- Pipeline dashboard → Metrics store: reads metrics (evidence: `lib/crunch_it/dashboard/pipeline_live.ex`)
- Pipeline dashboard → Kafka topic: fetches lag (evidence: `lib/crunch_it/dashboard/pipeline_live.ex`)
- Application supervision → Query lifecycle: starts service (evidence: `lib/crunch_it/application.ex`)
- Application supervision → Metrics store: starts service (evidence: `lib/crunch_it/application.ex`)
