How blackuprise/crunchit works
CrunchIt consumes Kafka messages through a query-specific pipeline, decodes and evaluates SQL filters/aggregations, tracks entity state, and periodically materializes aggregate views in a selected database. Queries are managed and activated through the configuration UI; a separate dashboard reports pipeline metrics. The README documents decoder choices, Redis deduplication, deletion, multiple storage adapters, and query lifecycle. Some implementation links are based on README descriptions because their source was not excerpted.
Main entry points: pipeline.ex Kafka pipeline · query_config.ex Query lifecycle · upsert_view.ex View updates · compiler.ex SQL compiler
Components 23
Query management
- Query UI
lib/crunch_it/dashboard/config_live.ex - Query lifecycle
lib/crunch_it/query_config.ex - Query definition
lib/crunch_it/domain/query.ex
Kafka ingestion
- Kafka pipeline
lib/crunch_it/infrastructure/kafka/pipeline.ex - Message decoders
lib/crunch_it/infrastructure/kafka/decoder.ex - Redis deduplication
lib/crunch_it/infrastructure/redis/dedup.ex
Query and aggregation
- SQL parser
lib/crunch_it/application/sql/parser.ex - SQL compiler
lib/crunch_it/application/sql/compiler.ex - SQL expressions
lib/crunch_it/application/sql/expression.ex - Compiled query
lib/crunch_it/domain/compiled_query.ex - SQL evaluator
lib/crunch_it/application/sql/evaluator.ex - Aggregation behaviors
lib/crunch_it/aggregation/incremental.ex
State and storage
- Window state
lib/crunch_it/infrastructure/kafka/ets_state.ex - Window flush
lib/crunch_it/infrastructure/kafka/flush_server.ex - View updates
lib/crunch_it/application/materialization/commands/upsert_view.ex - Postgres materialization
lib/crunch_it/infrastructure/postgres/upsert.ex - Storage adapters
lib/crunch_it/storage/adapter.ex - Associated ID tracking
lib/crunch_it/application/materialization/commands/store_ids.ex
Dashboard and operations
- Pipeline dashboard
lib/crunch_it/dashboard/pipeline_live.ex - Metrics store
lib/crunch_it/dashboard/metrics_store.ex - Application supervision
lib/crunch_it/application.ex
Other components
- Operator
- Kafka topic
Connections 24
- Operator → Query UI: manages queries
- Query UI → Query lifecycle: submits changes
- Query lifecycle → SQL compiler: compiles query
lib/crunch_it/query_config.ex - Query lifecycle → Compiled query: starts pipeline with
- Query definition → SQL parser: contains SQL
- SQL parser → SQL compiler: parses query for
- SQL compiler → SQL expressions: compiles expressions
- Compiled query → Kafka pipeline: configures
lib/crunch_it/infrastructure/kafka/pipeline.ex - Kafka topic → Kafka pipeline: provides messages
- Kafka pipeline → Message decoders: decodes messages
- Kafka pipeline → SQL evaluator: evaluates messages
lib/crunch_it/infrastructure/kafka/pipeline.ex - Kafka pipeline → Redis deduplication: checks and marks keys
lib/crunch_it/infrastructure/kafka/pipeline.ex - SQL evaluator → Aggregation behaviors: updates aggregates
- Kafka pipeline → Window state: reads and writes state
lib/crunch_it/infrastructure/kafka/pipeline.ex - Kafka pipeline → Window flush: requests flush
lib/crunch_it/infrastructure/kafka/pipeline.ex - Window flush → View updates: materializes window
- View updates → Postgres materialization: upserts view
- View updates → Storage adapters: uses selected backend
- Kafka pipeline → Associated ID tracking: tracks associated IDs
- Kafka pipeline → Metrics store: records metrics
lib/crunch_it/infrastructure/kafka/pipeline.ex - Pipeline dashboard → Metrics store: reads metrics
lib/crunch_it/dashboard/pipeline_live.ex - Pipeline dashboard → Kafka topic: fetches lag
lib/crunch_it/dashboard/pipeline_live.ex - Application supervision → Query lifecycle: starts service
lib/crunch_it/application.ex - Application supervision → Metrics store: starts service
lib/crunch_it/application.ex