# airbnb/chronon architecture

> Architecture diagram of the GitHub repository airbnb/chronon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/airbnb/chronon
- Repository: https://github.com/airbnb/chronon

## Overview

##### Project type and purpose
- Chronon is a full-stack data platform for AI/ML that provides:
– A Python/Scala API (SDK) to define “GroupBy” (windowed aggregations) and “Join” (feature backfill) transformations
– Batch compute via Spark for backfills and consistency pipelines
– Streaming compute via Flink for online/updating aggregations
– An online low-latency serving layer (Scala “online” module + Java/Vert.x “service”)
– Orchestration via Airflow DAGs and CLI tools (compile.py, run.py)
– A backing metadata catalog and key-value store (e.g. Hive/HDFS for tables, MongoDB for KV store)

##### File structure
- Top-level modules:
– aggregator/: core Scala library of aggregation primitives
– spark/: Spark batch jobs using aggregator library
– flink/: Flink streaming jobs using aggregator
– online/: Scala code for fetching feature values from KV store
– service/: Java/Vert.x web service exposing feature APIs
– api/: Python CLI/SDK to define, compile (Thrift), submit and fetch jobs
– airflow/: Python DAG constructors to orchestrate batch/stream workflows
– quickstart/: demo environment (Docker, Spark, Mongo)
- Build & deploy:
– Bazel/SBT build files, Dockerfiles, docker-compose.yml
– compile.py & run.py orchestrate Thrift generation and spark/flink job submission
– Airflow config in .circleci/config.yml and airflow/ directory

##### Key insights from README.md
- Users write GroupBy and Join definitions in Python
- compile.py → generates Thrift definitions
- run.py → submits backfill/upload/fetch/consistency tasks
- Online serving writes/reads from MongoDB KV store
- Observability via logging fetches and running consistency-check Spark jobs

##### Main components
a. SDK & CLI Layer
– Python SDK (api/py) and Scala API (api/src)
– CLI tools: compile.py, run.py
b. Orchestration Layer
– Airflow DAGs (airflow/)
– CICD pipelines (.circleci/, GitHub actions)
c. Batch Compute Plane
– Spark jobs (spark/)
– Hive/HDFS metastore for raw data & backfill output
– Thrift definitions repo (api/thrift)
d. Streaming Compute Plane
– Flink jobs (flink/)
– Kafka topics / event streams for real-time updates
– Aggregator library (aggregator/)
e. Serving & Storage Layer
– MongoDB or other KV store for online features
– Scala “online” fetcher (online/)
– Java/Vert.x web service (service/)
f. Observability & Consistency
– Logging dumper (service module or Spark job)
– Consistency-metrics Spark job via run.py

##### Relationships and interactions
1. User defines features in Python → passes GroupBy/Join configs to compile.py
2. compile.py generates Thrift schemas → artifacts stored in api/thrift
3. run.py submit jobs:
– Batch backfill: Spark ← Thrift definitions → writes wide tables to Hive
– Upload to KV: Spark → MongoDB
– Streaming updates: Flink jobs subscribe to Kafka, update KV
– Consistency check: Spark reads log table + re-backfills from Hive, compares to KV
4. Airflow DAGs orchestrate compile.py & run.py tasks on a schedule or event
5. Online service (Vert.x) loads fetcher & metadata, handles HTTP/GRPC client requests → reads MongoDB → returns feature vectors

##### Architectural patterns and design principles
- Separation of concerns: API/SDK, orchestrator, compute engines, serving
- Polyglot: Python for DSL, Scala for compute, Java for web service
- Micro-batches (Spark) + true streaming (Flink)
- Thrift as canonical schema between API definitions and compute
- Bazel/SBT for reproducible multi-language builds

##### Technologies & frameworks to call out
- Spark (batch backfill, consistency, backfill output)
- Flink (stream processing, real-time aggregator)
- Airflow (workflow orchestration)
- Scala/Java (aggregators, online fetcher, service)
- Python (SDK, CLI, DAG constructors)
- Thrift (schema exchange)
- MongoDB (online KV store)
- Hive/HDFS (batch tables, logs)
- Kafka (stream input)
- Vert.x (web service)

## Diagram (Mermaid)

```mermaid
graph TD
    %% SDK & CLI Layer
    subgraph "SDK & CLI Layer"
        PySDK["Python SDK (DSL & CLI)"]:::user
        ScalaSDK["Scala API (SDK)"]:::user
        Compile["compile.py"]:::user
        RunCLI["run.py"]:::user
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Airflow["Airflow DAGs"]:::orchestration
        CI["CI Config"]:::orchestration
    end

    %% Batch Compute Plane
    subgraph "Batch Compute Plane"
        Thrift["Thrift Schema Definitions"]:::compute
        Spark["Spark Jobs"]:::compute
        Consistency["Consistency Job"]:::compute
        LogFlatten["Log Flattener Job"]:::compute
        Hive["Hive/HDFS Metastore"]:::storage
    end

    %% Streaming Compute Plane
    subgraph "Streaming Compute Plane"
        Kafka["Kafka Topics"]:::storage
        Flink["Flink Jobs"]:::compute
        Aggregator["Aggregator Library"]:::compute
    end

    %% Serving & Storage Layer
    subgraph "Serving & Storage Layer"
        Mongo["MongoDB KV Store"]:::storage
        Fetcher["Scala Online Fetcher"]:::service
        WebService["Java/Vert.x Web Service"]:::service
    end

    %% Quickstart Environment
    subgraph "Quickstart Environment"
        Quickstart["Quickstart Demo"]:::demo
    end

    %% External Client
    Client["Client (HTTP/GRPC)"]:::client

    %% Relationships
    PySDK -->|"define features"| Compile
    ScalaSDK -->|"define features"| Compile
    Compile -->|"generate Thrift"| Thrift
    Airflow -->|"orchestrate compile"| Compile
    Airflow -->|"orchestrate run"| RunCLI
    CI -->|"CI pipelines"| Airflow

    RunCLI -->|"submit batch"| Spark
    RunCLI -->|"submit streaming"| Flink
    RunCLI -->|"run consistency"| Consistency

    Thrift -->|"schemas"| Spark
    Thrift -->|"schemas"| Flink

    Aggregator -->|"primitives used by"| Spark
    Aggregator -->|"primitives used by"| Flink

    Spark -->|"write wide tables"| Hive
    Spark -->|"upload features"| Mongo

    LogFlatten -->|"flatten logs"| Hive
    Hive -->|"log table"| Consistency
    Mongo -->|"compare KV"| Consistency

    Kafka -->|"stream events"| Flink
    Flink -->|"update features"| Mongo

    Client -->|"request features"| WebService
    WebService -->|"invoke fetcher"| Fetcher
    Fetcher -->|"read features"| Mongo
    WebService -->|"return vectors"| Client

    Quickstart -->|"demo env"| PySDK
    Quickstart -->|"demo env"| Spark
    Quickstart -->|"demo env"| Mongo

    %% Click Events
    click PySDK "https://github.com/airbnb/chronon/tree/main/api/py"
    click ScalaSDK "https://github.com/airbnb/chronon/tree/main/api/src/main/scala/ai/chronon/api"
    click Compile "https://github.com/airbnb/chronon/blob/main/api/py/ai/chronon/repo/compile.py"
    click RunCLI "https://github.com/airbnb/chronon/blob/main/api/py/ai/chronon/repo/run.py"
    click Airflow "https://github.com/airbnb/chronon/tree/main/airflow/"
    click CI "https://github.com/airbnb/chronon/blob/main/.circleci/config.yml"
    click Aggregator "https://github.com/airbnb/chronon/tree/main/aggregator/"
    click Spark "https://github.com/airbnb/chronon/tree/main/spark/"
    click Thrift "https://github.com/airbnb/chronon/blob/main/api/thrift/api.thrift"
    click Flink "https://github.com/airbnb/chronon/tree/main/flink/"
    click Fetcher "https://github.com/airbnb/chronon/tree/main/online/"
    click WebService "https://github.com/airbnb/chronon/tree/main/service/"
    click Consistency "https://github.com/airbnb/chronon/blob/main/spark/src/main/scala/ai/chronon/spark/stats/ConsistencyJob.scala"
    click LogFlatten "https://github.com/airbnb/chronon/blob/main/spark/src/main/scala/ai/chronon/spark/LogFlattenerJob.scala"
    click Quickstart "https://github.com/airbnb/chronon/tree/main/quickstart/"

    %% Styles
    classDef user fill:#D0E6FF,stroke:#1E70BF,color:#1E70BF
    classDef orchestration fill:#E4F7D2,stroke:#4C9A2A,color:#4C9A2A
    classDef compute fill:#FFD8B8,stroke:#D9822B,color:#D9822B
    classDef storage fill:#F0F0F0,stroke:#A0A0A0,color:#555555
    classDef service fill:#EAD7FF,stroke:#8A2BE2,color:#8A2BE2
    classDef demo fill:#FFF4D6,stroke:#E1A100,color:#E1A100
    classDef client fill:#FFFFFF,stroke:#000000,color:#000000,stroke-dasharray: 2 2
```
