# linkedin/venice architecture

> Architecture diagram of the GitHub repository linkedin/venice, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linkedin/venice
- Repository: https://github.com/linkedin/venice

## Overview

Venice is a derived-data storage platform. Its documented lifecycle takes batch or streaming writes through ingestion into partitioned local storage, then serves reads through remote clients or Da Vinci’s in-process cache; CDC exposes subsequent changes. The README also describes hybrid ingestion, write compute, active-active conflict resolution, and read compute. The tree and excerpts establish Da Vinci ingestion, storage metadata, blob transfer, controller, router dictionary retrieval, and a Fast Client tool. Several server-side and client subsystems are represented only by documented capabilities, so their internal wiring is intentionally not inferred.

Main entry points: [`clients/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/KafkaStoreIngestionService.java`](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/KafkaStoreIngestionService.java) (Partition Ingestion), [`clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageService.java`](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageService.java) (Storage Service), [`services/venice-router/src/main/java/com/linkedin/venice/router/api/DictionaryRetrievalService.java`](https://github.com/linkedin/venice/blob/main/services/venice-router/src/main/java/com/linkedin/venice/router/api/DictionaryRetrievalService.java) (Router), [`clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciClient.java`](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciClient.java) (Da Vinci Client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_write["Write and Ingestion"]
  node_ingestion["Partition Ingestion"]
  node_writecompute["Write Compute"]
  node_merge["Conflict Resolution"]
end

subgraph group_store["Local Data"]
  node_storage["Storage Service"]
  node_metadata["Storage Metadata"]
end

subgraph group_serve["Read and Serve"]
  node_davinci["Da Vinci Client<br/>[DaVinciClient.java]"]
  node_fastclient["Fast Client"]
  node_thinclient["Thin Client"]
  node_router["Router"]
  node_cdc["Change Consumer"]
  node_readcompute["Read Compute"]
end

subgraph group_operations["Control and Replication"]
  node_controller["Controller"]
  node_blob["Peer Blob Transfer"]
end

node_producer(("Batch/Stream Writers"))
node_pubsub["Pub/Sub Log"]
node_actor(("Applications"))
node_batch["Batch Systems"]
node_stream["Streaming Systems"]
node_regional["Peer Regions"]

node_batch -->|"submit batches"| node_producer
node_stream -->|"publish updates"| node_producer
node_producer -->|"write records"| node_pubsub
node_pubsub -->|"deliver records"| node_ingestion
node_ingestion -->|"apply updates"| node_writecompute
node_ingestion -->|"resolve conflicts"| node_merge
node_ingestion -->|"persist records"| node_storage
node_ingestion -->|"persist offsets"| node_metadata
node_actor -->|"read locally"| node_davinci
node_actor -->|"query remotely"| node_fastclient
node_actor -->|"query remotely"| node_thinclient
node_fastclient -->|"route queries"| node_router
node_thinclient -->|"route queries"| node_router
node_router -->|"serve reads"| node_storage
node_davinci -->|"read local data"| node_storage
node_actor -->|"subscribe changes"| node_cdc
node_cdc -->|"consume changes"| node_pubsub
node_actor -->|"request computation"| node_readcompute
node_controller -->|"manage ingestion"| node_ingestion
node_regional -.->|"request snapshots"| node_blob
node_blob -.->|"transfer snapshot"| node_storage

click node_ingestion "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/KafkaStoreIngestionService.java"
click node_writecompute "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/schema/writecompute/WriteComputeProcessor.java"
click node_merge "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/replication/merge/MergeConflictResolver.java"
click node_storage "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageService.java"
click node_metadata "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageEngineMetadataService.java"
click node_davinci "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciClient.java"
click node_fastclient "https://github.com/linkedin/venice/blob/main/clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java"
click node_thinclient "https://github.com/linkedin/venice/tree/main/clients/venice-client"
click node_router "https://github.com/linkedin/venice/blob/main/services/venice-router/src/main/java/com/linkedin/venice/router/api/DictionaryRetrievalService.java"
click node_cdc "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/consumer/VeniceChangelogConsumer.java"
click node_readcompute "https://github.com/linkedin/venice/tree/main/clients/venice-client"
click node_controller "https://github.com/linkedin/venice/blob/main/services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerService.java"
click node_blob "https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/blobtransfer/BlobTransferManager.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_writecompute,node_merge toneBlue
class node_storage,node_metadata toneAmber
class node_davinci,node_fastclient,node_thinclient,node_router,node_cdc,node_readcompute toneMint
class node_controller,node_blob toneRose
class node_producer,node_pubsub,node_actor,node_batch,node_stream,node_regional toneIndigo
```

## Components

### Write and Ingestion

- [Partition Ingestion](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/KafkaStoreIngestionService.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/KafkaStoreIngestionService.java`
- [Write Compute](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/schema/writecompute/WriteComputeProcessor.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/schema/writecompute/WriteComputeProcessor.java`
- [Conflict Resolution](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/replication/merge/MergeConflictResolver.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/replication/merge/MergeConflictResolver.java`

### Local Data

- [Storage Service](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageService.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageService.java`
- [Storage Metadata](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageEngineMetadataService.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageEngineMetadataService.java`

### Read and Serve

- [Da Vinci Client](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciClient.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciClient.java`
- [Fast Client](https://github.com/linkedin/venice/blob/main/clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java): `clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java`
- [Thin Client](https://github.com/linkedin/venice/tree/main/clients/venice-client): `clients/venice-client`
- [Router](https://github.com/linkedin/venice/blob/main/services/venice-router/src/main/java/com/linkedin/venice/router/api/DictionaryRetrievalService.java): `services/venice-router/src/main/java/com/linkedin/venice/router/api/DictionaryRetrievalService.java`
- [Change Consumer](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/consumer/VeniceChangelogConsumer.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/consumer/VeniceChangelogConsumer.java`
- [Read Compute](https://github.com/linkedin/venice/tree/main/clients/venice-client): `clients/venice-client`

### Control and Replication

- [Controller](https://github.com/linkedin/venice/blob/main/services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerService.java): `services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerService.java`
- [Peer Blob Transfer](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/blobtransfer/BlobTransferManager.java): `clients/da-vinci-client/src/main/java/com/linkedin/davinci/blobtransfer/BlobTransferManager.java`

### Other components

- **Batch/Stream Writers**
- **Pub/Sub Log**
- **Applications**
- **Batch Systems**
- **Streaming Systems**
- **Peer Regions**

## Connections

- Batch Systems → Batch/Stream Writers: submit batches
- Streaming Systems → Batch/Stream Writers: publish updates
- Batch/Stream Writers → Pub/Sub Log: write records
- Pub/Sub Log → Partition Ingestion: deliver records
- Partition Ingestion → Write Compute: apply updates
- Partition Ingestion → Conflict Resolution: resolve conflicts
- Partition Ingestion → Storage Service: persist records
- Partition Ingestion → Storage Metadata: persist offsets
- Applications → Da Vinci Client: read locally
- Applications → Fast Client: query remotely
- Applications → Thin Client: query remotely
- Fast Client → Router: route queries
- Thin Client → Router: route queries
- Router → Storage Service: serve reads
- Da Vinci Client → Storage Service: read local data
- Applications → Change Consumer: subscribe changes
- Change Consumer → Pub/Sub Log: consume changes
- Applications → Read Compute: request computation
- Controller → Partition Ingestion: manage ingestion
- Peer Regions → Peer Blob Transfer: request snapshots
- Peer Blob Transfer → Storage Service: transfer snapshot