How infinitelevel/data_sketches works
Project Type and Purpose
• This is a research-driven, distributed, streaming data-processing system.
• It consists of multiple Clients that ingest and sketch data, a central Server that merges sketches into a global state, and optional Consumers that query that state.
• There are also benchmarking and auxiliary command-line components for performance evaluation and automation.
Main Components
A. Data Sources
• CSV datasets under /data or generated by generate_zipf_data.py.
• (Optional) pcap files converted via script/pcap2csv.go.
B. Clients (/client + root main.go with -client flag)
• Implements streaming ingestion (stream/stream.go).
• Applies one of the pluggable sketch algorithms found in /sketches (KLL, Count Sketch, HLL, etc.).
• Periodically issues MergeRequests over gRPC (definitions in /proto/sketch.proto).
• Command-line flags control server address, sketch type, mergeRate, streamRate, dataset paths.
C. Server (/server + root main.go with no flags or -port)
• Hosts a gRPC service (sketch_grpc.pb.go) listening on a configurable port.
• Maintains an in-memory “global sketch state” per sketch type.
• Merges incoming client sketches into global state using the same algorithms.
D. Consumers (/consumer)
• Connects via gRPC to the Server.
• Issues queries (e.g., quantile or frequency queries) against the current global sketch.
• Provides an interactive REPL (“help” command).
E. Auxiliary Tools & Benchmarks
• /benchmarking: end-to-end scripts and plots for client-side, server-side, and system-level throughput/latency.
• /auto_query: automated consumer-style querying for CI or long-running tests.
• /cmd: helper binaries (e.g., runPinger, createHists) for network profiling or histogram creation.
Relationships and Data Flow
1. Data Ingestion → Client sketch pipeline → Local Sketch State
2. After N items (mergeRate), Client serializes local sketch and calls Server.MergeSketch() (gRPC).
3. Server updates its Global Sketch State.
4. Consumer calls Server.QuerySketch() (gRPC) → receives sketch summary or query result.
5. (Benchmarking) Scripts orchestrate multiple Clients/Server runs, collect logs, and feed into /plot scripts.
Architectural Patterns & Design Principles
• Strategy Pattern for pluggable sketch algorithms (/sketches/*).
• RPC-based microservice (gRPC) connecting Clients, Server, Consumers.
• Single responsibility: Clients only ingest & sketch; Server only merge & serve; Consumers only query.
• Separation of concerns: benchmarking code is isolated under /benchmarking; plotting under /plot.