How airbnb/chronon works
Project type and purpose
• Chronon is a full-stack data platform for AI/ML that provides:
– A Python/Scala API (SDK) to define “GroupBy” (windowed aggregations) and “Join” (feature backfill) transformations
– Batch compute via Spark for backfills and consistency pipelines
– Streaming compute via Flink for online/updating aggregations
– An online low-latency serving layer (Scala “online” module + Java/Vert.x “service”)
– Orchestration via Airflow DAGs and CLI tools (compile.py, run.py)
– A backing metadata catalog and key-value store (e.g. Hive/HDFS for tables, MongoDB for KV store)
File structure
• Top-level modules:
– aggregator/: core Scala library of aggregation primitives
– spark/: Spark batch jobs using aggregator library
– flink/: Flink streaming jobs using aggregator
– online/: Scala code for fetching feature values from KV store
– service/: Java/Vert.x web service exposing feature APIs
– api/: Python CLI/SDK to define, compile (Thrift), submit and fetch jobs
– airflow/: Python DAG constructors to orchestrate batch/stream workflows
– quickstart/: demo environment (Docker, Spark, Mongo)
• Build & deploy:
– Bazel/SBT build files, Dockerfiles, docker-compose.yml
– compile.py & run.py orchestrate Thrift generation and spark/flink job submission
– Airflow config in .circleci/config.yml and airflow/ directory
Key insights from README.md
• Users write GroupBy and Join definitions in Python
• compile.py → generates Thrift definitions
• run.py → submits backfill/upload/fetch/consistency tasks
• Online serving writes/reads from MongoDB KV store
• Observability via logging fetches and running consistency-check Spark jobs
Main components
a. SDK & CLI Layer
– Python SDK (api/py) and Scala API (api/src)
– CLI tools: compile.py, run.py
b. Orchestration Layer
– Airflow DAGs (airflow/)
– CICD pipelines (.circleci/, GitHub actions)
c. Batch Compute Plane
– Spark jobs (spark/)
– Hive/HDFS metastore for raw data & backfill output
– Thrift definitions repo (api/thrift)
d. Streaming Compute Plane
– Flink jobs (flink/)
– Kafka topics / event streams for real-time updates
– Aggregator library (aggregator/)
e. Serving & Storage Layer
– MongoDB or other KV store for online features
– Scala “online” fetcher (online/)
– Java/Vert.x web service (service/)
f. Observability & Consistency
– Logging dumper (service module or Spark job)
– Consistency-metrics Spark job via run.py
Relationships and interactions
1. User defines features in Python → passes GroupBy/Join configs to compile.py
2. compile.py generates Thrift schemas → artifacts stored in api/thrift
3. run.py submit jobs:
– Batch backfill: Spark ← Thrift definitions → writes wide tables to Hive
– Upload to KV: Spark → MongoDB
– Streaming updates: Flink jobs subscribe to Kafka, update KV
– Consistency check: Spark reads log table + re-backfills from Hive, compares to KV
4. Airflow DAGs orchestrate compile.py & run.py tasks on a schedule or event
5. Online service (Vert.x) loads fetcher & metadata, handles HTTP/GRPC client requests → reads MongoDB → returns feature vectors
Architectural patterns and design principles
• Separation of concerns: API/SDK, orchestrator, compute engines, serving
• Polyglot: Python for DSL, Scala for compute, Java for web service
• Micro-batches (Spark) + true streaming (Flink)
• Thrift as canonical schema between API definitions and compute
• Bazel/SBT for reproducible multi-language builds
Technologies & frameworks to call out
• Spark (batch backfill, consistency, backfill output)
• Flink (stream processing, real-time aggregator)
• Airflow (workflow orchestration)
• Scala/Java (aggregators, online fetcher, service)
• Python (SDK, CLI, DAG constructors)
• Thrift (schema exchange)
• MongoDB (online KV store)
• Hive/HDFS (batch tables, logs)
• Kafka (stream input)
• Vert.x (web service)