How unmeshjoshi/ds-patterns-workshop works
This repository is a workshop collection of runnable, laptop-scale models of Cassandra, Delta Lake, Kafka, MinIO-style object storage, Parquet, Spark, and Kubernetes. There is no single cross-repository user workflow: each demo models a system-specific path from client or controller activity to a stored, scheduled, or replicated result. The graph preserves the documented systems and their major domain stages. Internal edges are limited to relationships visible in excerpts; the remaining subsystem boundaries are based on the README and file tree, not unsampled call paths.
Main entry points: DeltaLog.java Transaction log · StorageNode.java Storage node · ObjectStoreParquetReader.java Parquet reader · ReplicaManager.java Replica manager
Components 34
Cassandra model
- Gossip state
cassandralite/src/main/java/cassandralite/gossip/Gossiper.java - Failure detection
cassandralite/src/main/java/cassandralite/heartbeat/FailureDetector.java - Cluster protocol
cassandralite/src/main/java/cassandralite/protocol/CassandraLiteProtocol.java
Delta Lake model
- Table operations
deltalite/step3-snapshot-occ/src/main/java/deltalite/DeltaTable.java - Transaction log
deltalite/step3-snapshot-occ/src/main/java/deltalite/DeltaLog.java - Snapshot state
deltalite/step3-snapshot-occ/src/main/java/deltalite/Snapshot.java - Optimistic transaction
deltalite/step3-snapshot-occ/src/main/java/deltalite/OptimisticTransaction.java - Log actions
deltalite/step3-snapshot-occ/src/main/java/deltalite/actions/Action.java - Object storage adapter
deltalite/step4-object-store/src/main/java/deltalite/store/ObjectStoreStorage.java - Log file naming
deltalite/step2-commit-log/src/main/java/deltalite/util/FileNames.java - Log serialization
deltalite/step2-commit-log/src/main/java/deltalite/util/JsonUtil.java
Kafka model
- Consumer client
kafkalite/src/main/java/kafkalite/client/KafkaConsumerLite.java - Cluster controller
kafkalite/src/main/java/kafkalite/controller/ZkController.java - Controller channels
kafkalite/src/main/java/kafkalite/controller/ControllerChannelManager.java - ZooKeeper state
kafkalite/src/main/java/kafkalite/zookeeper/ZookeeperClient.java - Replica manager
kafkalite/src/main/java/kafkalite/cluster/ReplicaManager.java - Partition state
kafkalite/src/main/java/kafkalite/partition/Partition.java - Broker log
kafkalite/src/main/java/kafkalite/log/Log.java - Message record
kafkalite/src/main/java/kafkalite/api/Message.java
Object storage model
- Object-store client
objectstorelite/src/main/java/objectstorelite/client/ObjectStoreClient.java - Storage node
objectstorelite/src/main/java/objectstorelite/server/StorageNode.java - Object-store protocol
objectstorelite/src/main/java/objectstorelite/protocol/ObjectStoreProtocol.java - Erasure coding
objectstorelite/src/main/java/objectstorelite/codec/ErasureCodec.java - Shard files
objectstorelite/src/main/java/objectstorelite/storage/AtomicFiles.java
Parquet model
- Parquet reader
parquetlite/src/main/java/parquetlite/file/ObjectStoreParquetReader.java - Footer reader
parquetlite/src/main/java/parquetlite/file/ParquetFooterReader.java - Row-group filter
parquetlite/src/main/java/parquetlite/filter/RowGroupFilter.java - Range input
parquetlite/src/main/java/parquetlite/io/PrefetchedInputFile.java
Spark model
- Spark context
sparklite/src/main/java/sparklite/context/SparkLiteContext.java - RDD lineage
sparklite/src/main/java/sparklite/rdd/RDDLite.java - DAG scheduler
sparklite/src/main/java/sparklite/scheduler/DAGScheduler.java - Task scheduler
sparklite/src/main/java/sparklite/scheduler/TaskScheduler.java - RDD task
sparklite/src/main/java/sparklite/scheduler/RDDLiteTask.java - Worker execution
sparklite/src/main/java/sparklite/worker/SparkLiteWorker.java
Connections 25
- Cluster controller → ZooKeeper state: reads cluster state
kafkalite/src/main/java/kafkalite/controller/ZkController.java - Cluster controller → Controller channels: sends assignments
kafkalite/src/main/java/kafkalite/controller/ZkController.java - Controller channels → Replica manager: dispatches requests
kafkalite/src/main/java/kafkalite/controller/ControllerChannelManager.java - Replica manager → Partition state: updates and reads
kafkalite/src/main/java/kafkalite/cluster/ReplicaManager.java - Partition state → Broker log: uses log
- Replica manager → Message record: appends and reads
kafkalite/src/main/java/kafkalite/cluster/ReplicaManager.java - Object-store client → Storage node: sends requests
- Storage node → Object-store protocol: handles messages
objectstorelite/src/main/java/objectstorelite/server/StorageNode.java - Storage node → Erasure coding: encodes objects
objectstorelite/src/main/java/objectstorelite/server/StorageNode.java - Storage node → Shard files: writes shards
objectstorelite/src/main/java/objectstorelite/server/StorageNode.java - Object storage adapter → Object-store client: uses client
deltalite/step4-object-store/src/main/java/deltalite/store/ObjectStoreStorage.java - Table operations → Transaction log: table history
- Transaction log → Snapshot state: materializes state
- Transaction log → Optimistic transaction: coordinates commits
- Transaction log → Log actions: records actions
- Transaction log → Log file naming: names log files
deltalite/step3-snapshot-occ/src/main/java/deltalite/DeltaLog.java - Transaction log → Log serialization: serializes entries
deltalite/step3-snapshot-occ/src/main/java/deltalite/DeltaLog.java - Parquet reader → Footer reader: reads footer
- Parquet reader → Row-group filter: filters groups
- Parquet reader → Range input: reads ranges
- Spark context → RDD lineage: builds lineage
- Spark context → DAG scheduler: submits jobs
- DAG scheduler → Task scheduler: schedules tasks
- Task scheduler → RDD task: serializes tasks
sparklite/src/main/java/sparklite/scheduler/TaskScheduler.java - Worker execution → RDD task: executes task
sparklite/src/main/java/sparklite/worker/SparkLiteWorker.java