How gojek/ziggurat works
Project’s Nature and Purpose
• Ziggurat is a stream processing framework intended for building applications that consume messages from Kafka.
• It abstracts common functionalities such as consuming messages, handling retries via RabbitMQ, and starting an HTTP server for additional features.
• The framework supports multiple streams and offers middleware integration (such as message serialization and metric reporting).
Main Components
a. External Services:
• Kafka Cluster: Represent the Kafka brokers and ZooKeeper (as seen with docker-compose files and Kafka configuration files).
• RabbitMQ: Used for handling retries for failed messages.
• HTTP Service: An embedded HTTP server that may expose APIs (as suggested by the HTTP server docs and route definitions).
b. Core Framework Layers:
• Kafka Consumer Module: This is represented by components in src/ziggurat/kafka_consumer. It handles reading messages from Kafka.
• Message Handler & Middleware: The core message-processing logic (main-fn and middleware functions such as protobuf->hash transformation) is located in src/ziggurat.
• Messaging Utilities: Components dealing with both Kafka and RabbitMQ messaging (visible in src/ziggurat/messaging and src/com/gojek/rabbitmq). Also include channel pools and connection helpers.
• Metrics Module: Located in src/com/gojek/metrics and src/ziggurat/metrics, integrating with Datadog, Prometheus exporters, and Dropwizard.
• Configuration and Bootstrapping: The configuration files (such as config-admin.properties, various EDN files) and initialization scripts (src/ziggurat/init.clj) that set up the system.
• Testing and Build Scripts: Although not part of the runtime architecture, the presence of extensive tests, Makefile, and docker-compose setups indicate robust deployment/testing practices.
Relationships and Interactions
• Data Flow from Kafka to Application:
• The consumer calls the registered handler (main-fn) with message payloads.
• Middleware Processing:
• After consumption, messages pass through a series of middleware functions (e.g., for serialization/deserialization and metrics tagging).
• Retry Mechanism:
• HTTP Server Integration:
• The framework also starts an HTTP server to allow status checks, additional resource endpoints, or even dynamic stream-route toggling. Link a component for “HTTP Server” which interacts with the core application.
• Metrics Collection:
Architectural Patterns and Design Principles
• Modular Design: The separation into distinct modules (Kafka consumer, messaging, middleware, HTTP server) suggests a modular architecture with clear separation of concerns.
• Extensibility & Multiple Streams: Since Ziggurat supports multiple stream routes, consider indicating that the system can handle parallel processing with separate pipelines identified by distinct stream IDs.
Technologies, Frameworks, and Libraries
• Clojure: The main language powering the framework (core components in src/ziggurat).
• Java: Certain metric reporting components are implemented in Java under src/com/gojek/metrics/datadog.
• Kafka, RabbitMQ, and ZooKeeper: External message brokers and coordination services.
• Docker Compose: For local development and cluster setup.
• Leiningen/Project.clj: For dependency management and building the project.
• Datadog, Prometheus, Dropwizard: Metrics and monitoring integrations.