How linkedin/venice works
Venice is a derived-data storage platform. Its documented lifecycle takes batch or streaming writes through ingestion into partitioned local storage, then serves reads through remote clients or Da Vinci’s in-process cache; CDC exposes subsequent changes. The README also describes hybrid ingestion, write compute, active-active conflict resolution, and read compute. The tree and excerpts establish Da Vinci ingestion, storage metadata, blob transfer, controller, router dictionary retrieval, and a Fast Client tool. Several server-side and client subsystems are represented only by documented capabilities, so their internal wiring is intentionally not inferred.
Main entry points: KafkaStoreIngestionService.java Partition Ingestion · StorageService.java Storage Service · DictionaryRetrievalService.java Router · DaVinciClient.java Da Vinci Client
Components 19
Write and Ingestion
- Partition Ingestion
clients/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/KafkaStoreIngestionService.java - Write Compute
clients/da-vinci-client/src/main/java/com/linkedin/davinci/schema/writecompute/WriteComputeProcessor.java - Conflict Resolution
clients/da-vinci-client/src/main/java/com/linkedin/davinci/replication/merge/MergeConflictResolver.java
Local Data
- Storage Service
clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageService.java - Storage Metadata
clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/StorageEngineMetadataService.java
Read and Serve
- Da Vinci Client
clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciClient.java - Fast Client
clients/venice-client/src/main/java/com/linkedin/venice/fastclient/FastClientQueryTool.java - Thin Client
clients/venice-client - Router
services/venice-router/src/main/java/com/linkedin/venice/router/api/DictionaryRetrievalService.java - Change Consumer
clients/da-vinci-client/src/main/java/com/linkedin/davinci/consumer/VeniceChangelogConsumer.java - Read Compute
clients/venice-client
Control and Replication
- Controller
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerService.java - Peer Blob Transfer
clients/da-vinci-client/src/main/java/com/linkedin/davinci/blobtransfer/BlobTransferManager.java
Other components
- Batch/Stream Writers
- Pub/Sub Log
- Applications
- Batch Systems
- Streaming Systems
- Peer Regions
Connections 21
- Batch Systems → Batch/Stream Writers: submit batches
- Streaming Systems → Batch/Stream Writers: publish updates
- Batch/Stream Writers → Pub/Sub Log: write records
- Pub/Sub Log → Partition Ingestion: deliver records
- Partition Ingestion → Write Compute: apply updates
- Partition Ingestion → Conflict Resolution: resolve conflicts
- Partition Ingestion → Storage Service: persist records
- Partition Ingestion → Storage Metadata: persist offsets
- Applications → Da Vinci Client: read locally
- Applications → Fast Client: query remotely
- Applications → Thin Client: query remotely
- Fast Client → Router: route queries
- Thin Client → Router: route queries
- Router → Storage Service: serve reads
- Da Vinci Client → Storage Service: read local data
- Applications → Change Consumer: subscribe changes
- Change Consumer → Pub/Sub Log: consume changes
- Applications → Read Compute: request computation
- Controller → Partition Ingestion: manage ingestion
- Peer Regions → Peer Blob Transfer: request snapshots
- Peer Blob Transfer → Storage Service: transfer snapshot