# infinitelevel/data_sketches architecture

> Architecture diagram of the GitHub repository infinitelevel/data_sketches, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/infinitelevel/data_sketches
- Repository: https://github.com/infinitelevel/data_sketches

## Overview

##### Project Type and Purpose
- This is a research-driven, distributed, streaming data-processing system.
- It consists of multiple Clients that ingest and sketch data, a central Server that merges sketches into a global state, and optional Consumers that query that state.
- There are also benchmarking and auxiliary command-line components for performance evaluation and automation.

##### Main Components
A. Data Sources
- CSV datasets under `/data` or generated by `generate_zipf_data.py`.
- (Optional) pcap files converted via `script/pcap2csv.go`.
B. Clients (`/client` + root `main.go` with `-client` flag)
- Implements streaming ingestion (`stream/stream.go`).
- Applies one of the pluggable sketch algorithms found in `/sketches` (KLL, Count Sketch, HLL, etc.).
- Periodically issues MergeRequests over gRPC (definitions in `/proto/sketch.proto`).
- Command-line flags control server address, sketch type, mergeRate, streamRate, dataset paths.
C. Server (`/server` + root `main.go` with no flags or `-port`)
- Hosts a gRPC service (`sketch_grpc.pb.go`) listening on a configurable port.
- Maintains an in-memory “global sketch state” per sketch type.
- Merges incoming client sketches into global state using the same algorithms.
D. Consumers (`/consumer`)
- Connects via gRPC to the Server.
- Issues queries (e.g., quantile or frequency queries) against the current global sketch.
- Provides an interactive REPL (“help” command).
E. Auxiliary Tools & Benchmarks
- `/benchmarking`: end-to-end scripts and plots for client-side, server-side, and system-level throughput/latency.
- `/auto_query`: automated consumer-style querying for CI or long-running tests.
- `/cmd`: helper binaries (e.g., `runPinger`, `createHists`) for network profiling or histogram creation.

##### Relationships and Data Flow
1. Data Ingestion → Client sketch pipeline → Local Sketch State
2. After N items (`mergeRate`), Client serializes local sketch and calls Server.MergeSketch() (gRPC).
3. Server updates its Global Sketch State.
4. Consumer calls Server.QuerySketch() (gRPC) → receives sketch summary or query result.
5. (Benchmarking) Scripts orchestrate multiple Clients/Server runs, collect logs, and feed into `/plot` scripts.

##### Architectural Patterns & Design Principles
- Strategy Pattern for pluggable sketch algorithms (`/sketches/*`).
- RPC-based microservice (gRPC) connecting Clients, Server, Consumers.
- Single responsibility: Clients only ingest & sketch; Server only merge & serve; Consumers only query.
- Separation of concerns: benchmarking code is isolated under `/benchmarking`; plotting under `/plot`.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        DS1["CSV Dataset"]:::data
        DS2["Generated Data"]:::data
        DS3["PCAP to CSV Script"]:::data
    end

    %% Clients
    subgraph "Clients" 
        CLApp["Client CLI"]:::ingestion
        Stream["Stream Reader"]:::ingestion
        SketchMod["Sketch Module"]:::algorithms
    end

    %% Sketch Algorithms
    subgraph "Sketch Algorithms"
        SA["Pluggable Algorithms"]:::algorithms
    end

    %% gRPC Definitions
    GRPCDefs["gRPC Definitions\n(proto/sketch.proto,\n proto/sketch.pb.go)"]:::core

    %% Server
    subgraph "Server"
        SVApp["Server App"]:::core
        GRPCSvc["gRPC Service"]:::core
        GlobalState["Global Sketch State"]:::core
        Persistence["Persistence\n(database)"]:::data
    end

    %% Consumers
    subgraph "Consumers"
        CUApp["Consumer CLI"]:::ingestion
        SharedTypes["Shared Types"]:::core
        GRPCCli["gRPC Client"]:::core
    end

    %% Tooling & Benchmarking
    subgraph "Tooling & Benchmarking"
        BM["Benchmarking"]:::tooling
        AQ["Automated Query"]:::tooling
        CLItools["Helper CLI Tools"]:::tooling
        Plotting["Plotting Scripts"]:::tooling
    end

    %% Connections
    DS1 -->|Ingest| CLApp
    DS2 -->|Ingest| CLApp
    DS3 -->|Ingest| CLApp

    CLApp -->|"reads stream"| Stream
    Stream -->|"uses"| SketchMod
    SketchMod -->|"implements"| SA

    SA -->|"defined in"| GRPCDefs

    CLApp -->|"MergeSketch()"| GRPCSvc
    GRPCSvc -->|"updates"| GlobalState
    GlobalState -->|"optionally writes"| Persistence

    CUApp -->|"uses Shared Types"| SharedTypes
    CUApp -->|"QuerySketch()"| GRPCCli
    GRPCCli -->|"calls"| GRPCSvc
    GRPCCli -->|"fetches"| GlobalState

    BM -->|"orchestrates"| CLApp
    BM -->|"orchestrates"| SVApp
    BM -->|"outputs data to"| Plotting

    AQ -->|"queries"| GRPCCli

    CLItools -->|"profiles"| CLApp
    CLItools -->|"profiles"| SVApp

    Plotting -->|"visualizes"| BM

    %% Click Events
    click DS1 "https://github.com/infinitelevel/data_sketches/blob/main/data/test_zipf.csv"
    click DS2 "https://github.com/infinitelevel/data_sketches/blob/main/generate_zipf_data.py"
    click DS3 "https://github.com/infinitelevel/data_sketches/blob/main/script/pcap2csv.go"
    click CLApp "https://github.com/infinitelevel/data_sketches/blob/main/client/client.go"
    click Stream "https://github.com/infinitelevel/data_sketches/blob/main/stream/stream.go"
    click SA "https://github.com/infinitelevel/data_sketches/tree/main/sketches"
    click GRPCDefs "https://github.com/infinitelevel/data_sketches/blob/main/proto/sketch.proto"
    click GRPCDefs "https://github.com/infinitelevel/data_sketches/blob/main/proto/sketch.pb.go"
    click SVApp "https://github.com/infinitelevel/data_sketches/blob/main/server/server.go"
    click GRPCSvc "https://github.com/infinitelevel/data_sketches/blob/main/proto/sketch_grpc.pb.go"
    click Persistence "https://github.com/infinitelevel/data_sketches/blob/main/database/.gitkeep"
    click CUApp "https://github.com/infinitelevel/data_sketches/blob/main/consumer/consumer.go"
    click SharedTypes "https://github.com/infinitelevel/data_sketches/blob/main/shared/types.go"
    click GRPCCli "https://github.com/infinitelevel/data_sketches/blob/main/proto/sketch_grpc.pb.go"
    click BM "https://github.com/infinitelevel/data_sketches/tree/main/benchmarking/"
    click AQ "https://github.com/infinitelevel/data_sketches/blob/main/auto_query/main.go"
    click CLItools "https://github.com/infinitelevel/data_sketches/blob/main/cmd/createHists/main.go"
    click CLItools "https://github.com/infinitelevel/data_sketches/blob/main/cmd/runPinger/main.go"
    click Plotting "https://github.com/infinitelevel/data_sketches/blob/main/plot/plot_client_throughput.py"
    click Plotting "https://github.com/infinitelevel/data_sketches/blob/main/plot/plot-system-throughput.py"
    click Plotting "https://github.com/infinitelevel/data_sketches/blob/main/plot/plot-server-latency.py"
    click Plotting "https://github.com/infinitelevel/data_sketches/blob/main/plot/plot_hist_from_data.go"
    click Plotting "https://github.com/infinitelevel/data_sketches/blob/main/plot_data.py"

    %% Styles
    classDef ingestion fill:#e0ffe0,stroke:#333,stroke-width:1px
    classDef core fill:#e0f0ff,stroke:#333,stroke-width:1px
    classDef algorithms fill:#ffe080,stroke:#333,stroke-width:1px
    classDef data fill:#f0e0e0,stroke:#333,stroke-width:1px
    classDef tooling fill:#e0e0e0,stroke:#333,stroke-width:1px
```
