# gojek/ziggurat architecture

> Architecture diagram of the GitHub repository gojek/ziggurat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gojek/ziggurat
- Repository: https://github.com/gojek/ziggurat

## Overview

##### Project’s Nature and Purpose
- Ziggurat is a stream processing framework intended for building applications that consume messages from Kafka.
- It abstracts common functionalities such as consuming messages, handling retries via RabbitMQ, and starting an HTTP server for additional features.
- The framework supports multiple streams and offers middleware integration (such as message serialization and metric reporting).

##### Main Components
a. External Services:
- Kafka Cluster: Represent the Kafka brokers and ZooKeeper (as seen with docker-compose files and Kafka configuration files).
- RabbitMQ: Used for handling retries for failed messages.
- HTTP Service: An embedded HTTP server that may expose APIs (as suggested by the HTTP server docs and route definitions).
b. Core Framework Layers:
- Kafka Consumer Module: This is represented by components in src/ziggurat/kafka_consumer. It handles reading messages from Kafka.
- Message Handler & Middleware: The core message-processing logic (main-fn and middleware functions such as protobuf->hash transformation) is located in src/ziggurat.
- Messaging Utilities: Components dealing with both Kafka and RabbitMQ messaging (visible in src/ziggurat/messaging and src/com/gojek/rabbitmq). Also include channel pools and connection helpers.
- Metrics Module: Located in src/com/gojek/metrics and src/ziggurat/metrics, integrating with Datadog, Prometheus exporters, and Dropwizard.
- Configuration and Bootstrapping: The configuration files (such as config-admin.properties, various EDN files) and initialization scripts (src/ziggurat/init.clj) that set up the system.
- Testing and Build Scripts: Although not part of the runtime architecture, the presence of extensive tests, Makefile, and docker-compose setups indicate robust deployment/testing practices.

##### Relationships and Interactions
- Data Flow from Kafka to Application:
- The consumer calls the registered handler (main-fn) with message payloads.
- Middleware Processing:
- After consumption, messages pass through a series of middleware functions (e.g., for serialization/deserialization and metrics tagging).
- Retry Mechanism:
- HTTP Server Integration:
- The framework also starts an HTTP server to allow status checks, additional resource endpoints, or even dynamic stream-route toggling. Link a component for “HTTP Server” which interacts with the core application.
- Metrics Collection:

##### Architectural Patterns and Design Principles
- Modular Design: The separation into distinct modules (Kafka consumer, messaging, middleware, HTTP server) suggests a modular architecture with clear separation of concerns.
- Extensibility & Multiple Streams: Since Ziggurat supports multiple stream routes, consider indicating that the system can handle parallel processing with separate pipelines identified by distinct stream IDs.

##### Technologies, Frameworks, and Libraries
- Clojure: The main language powering the framework (core components in src/ziggurat).
- Java: Certain metric reporting components are implemented in Java under src/com/gojek/metrics/datadog.
- Kafka, RabbitMQ, and ZooKeeper: External message brokers and coordination services.
- Docker Compose: For local development and cluster setup.
- Leiningen/Project.clj: For dependency management and building the project.
- Datadog, Prometheus, Dropwizard: Metrics and monitoring integrations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        A["Kafka Cluster & ZooKeeper"]
        B["RabbitMQ"]
        C["Monitoring Tools (Datadog/Prometheus/Dropwizard)"]
        D["Metrics Module (Java)"]
    end

    %% Application Core Modules
    subgraph "Application Core"
        E["Kafka Consumer Module"]
        F["Core Message Handler"]
        G["Middleware"]
        H["Messaging Utilities"]
        I["HTTP Server"]
        J["Metrics Module (Clojure)"]
    end

    %% Infrastructure/Configuration
    subgraph "Infrastructure/Configuration"
        K["Configuration/Bootstrapping"]
        L["Testing/Build Scripts"]
    end

    %% Connections between External Systems and Application Core
    A -->|"provides"| E
    E -->|"calls"| F
    F -->|"processes"| G
    G -->|"transfersData"| H
    H -->|"routesRetries"| B

    %% Metrics Reporting Connections
    F -->|"reportsMetrics"| J
    J -->|"sendsTo"| C
    F -->|"reportsMetrics"| D
    D -->|"integratesWith"| C

    %% HTTP Server Interaction
    I -->|"handlesRoutes"| F
    F -->|"accessibleVia"| I

    %% Configuration Connection
    K -->|"configures"| F

    %% Styles for color coding
    class A,B,C,D external
    class E,F,G,H,I,J core
    class K,L infra

    classDef external fill:#FFD1DC,stroke:#C70039,stroke-width:2px;
    classDef core fill:#D1FFD6,stroke:#2ECC40,stroke-width:2px;
    classDef infra fill:#D6E5FF,stroke:#3498DB,stroke-width:2px;

    %% Click Events
    click A "https://github.com/gojek/ziggurat/blob/master/docker-compose.yml"
    click B "https://github.com/gojek/ziggurat/tree/master/src/com/gojek/rabbitmq"
    click D "https://github.com/gojek/ziggurat/tree/master/src/com/gojek/metrics"
    click E "https://github.com/gojek/ziggurat/tree/master/src/ziggurat/kafka_consumer"
    click F "https://github.com/gojek/ziggurat/blob/master/src/ziggurat/init.clj"
    click G "https://github.com/gojek/ziggurat/tree/master/src/ziggurat/middleware"
    click H "https://github.com/gojek/ziggurat/tree/master/src/ziggurat/messaging"
    click I "https://github.com/gojek/ziggurat/tree/master/src/ziggurat/server"
    click J "https://github.com/gojek/ziggurat/blob/master/src/ziggurat/metrics.clj"
    click K "https://github.com/gojek/ziggurat/blob/master/config-admin.properties"
    click L "https://github.com/gojek/ziggurat/tree/master/Makefile"
```
