# dynatrace-oss/hash4j architecture

> Architecture diagram of the GitHub repository dynatrace-oss/hash4j, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dynatrace-oss/hash4j
- Repository: https://github.com/dynatrace-oss/hash4j

## Overview

Based on our analysis of the project’s file tree and README file, we understand that hash4j is a Java library providing a wide range of non‐cryptographic hash algorithms (like Murmur3, Wyhash, Komihash, FarmHash, PolymurHash, XXH3), in addition to other functionalities such as approximate distinct counting (HyperLogLog, UltraLogLog, Martingale Estimator), similarity hashing (MinHash, SuperMinHash, SimHash, FastSimHash), file hashing (Imohash), and consistent hashing (JumpHash, JumpBackHash, and improved consistent weighted sampling). In other words, the project is structured as a library that is heavily modularized by its functional areas.

##### Project Type and Purpose
- The project is a Java-based hash library designed to offer multiple standalone hash functions and related data structures. It is intended for users who need high-quality hashing for purposes such as data deduplication, file comparison, distribution via consistent hashing, similarity assessment, and approximate distinct counting.
- The README clearly explains the features and lists usage examples in each domain (hashing algorithms, distinct counting, similarity hashing, file hashing, and consistent hashing).

##### File Structure
- Notice that the core Java source code is under the "src/main/java/com/dynatrace/hash4j" package. This package is further subdivided into clearly defined sub-packages like:
- consistent – Implementing different consistent hashing algorithms.
- distinctcount – Handling approximate distinct counting with HyperLogLog and UltraLogLog.
- file – Providing file-based hashing (Imohash).
- hashing – Containing all different hash function implementations (Murmur, Komihash, FarmHash, etc.).
- random – Offering utilities for pseudo-random number generation which support some hashing algorithms.
- similarity – Implementing similarity hash algorithms.
- util – Common utility classes.
- There are also extensive test suites under "src/test", benchmark modules in "src/jmh", and reference implementations in C++/Go in the "reference-implementations" directory.
- Build, workflow, and configuration files (Gradle scripts, GitHub workflows) indicate standard build management and CI integration.

##### Main Components and Their Relationships
a. Core Library Modules:
- "Hashing Module": Contains multiple implementations of hash functions, including separate classes for 32-bit, 64-bit, and 128-bit hashing.
- "Distinct Counting Module": Implements approximate counting algorithms (HyperLogLog, UltraLogLog, Martingale estimator) with facilities for merging and downsizing.
- "Similarity Hashing Module": Implements algorithms such as MinHash, SuperMinHash, and SimHash that compute signature vectors for estimating set similarity.
- "File Hashing Module": Implements Imohash that hashes files by sampling portions (beginning, middle, end).
- "Consistent Hashing Module": Implements consistent bucket assignment algorithms (JumpHash, JumpBackHash) to minimize reassignment effects when bucket counts change.
- "Random Utilities": Provides underlying pseudo-random generators integral to some hash calculations.
- "Utility Functions": Includes helper and common functions for preconditions, array packing, etc.

b. Testing & Benchmarking:
- Unit testing is represented by the "src/test" directory.
- Performance benchmarks via JMH reside in "src/jmh".
- These ensure that each hashing algorithm, estimator, and utility function meets quality and performance standards.

c. External/Reference Implementations:
- The "reference-implementations" directory contains native versions (C++, Go, etc.) for cross-checking and benchmarking the Java implementations.
- The "python" scripts are used for benchmarking evaluation and simulation.

##### Architectural Patterns and Design Principles
- Modular Design: Each functional component is divided into its own package, making the library extensible.
- Separation of Concerns: Hashing algorithms, distinct-count estimators, file hashing, and consistent hashing are clearly separated, allowing independent evolution.
- Test-driven: A rigorous testing layer in "src/test" and benchmarks in "src/jmh" show that performance and correctness are central concerns.
- Integration with Build Tools: Use of Gradle and CI workflows (GitHub Actions) indicates a modern continuous integration and delivery process.

## Diagram (Mermaid)

```mermaid
flowchart TD
    H["Hash4j Library"]:::core

    subgraph "Core Modules"
        A["Hashing Algorithms"]:::module
        B["Distinct Counting"]:::module
        C["Similarity Hashing"]:::module
        D["File Hashing (Imohash)"]:::module
        E["Consistent Hashing"]:::module
        F["Random Utilities"]:::module
        G["Utility Functions"]:::module
    end

    subgraph "Auxiliary Modules"
        T["Testing"]:::aux
        BM["Benchmarking"]:::aux
        X["External/Reference Implementations"]:::external
    end

    H -->|"uses"| A
    H -->|"uses"| B
    H -->|"uses"| C
    H -->|"uses"| D
    H -->|"uses"| E
    H -->|"uses"| F
    H -->|"uses"| G
    H -->|"validated_by"| T
    H -->|"benchmarked_by"| BM
    H -->|"verified_by"| X

    E -->|"uses_Random"| F
    B -->|"merges_utilities"| G

    click A "https://github.com/dynatrace-oss/hash4j/tree/main/src/main/java/com/dynatrace/hash4j/hashing"
    click B "https://github.com/dynatrace-oss/hash4j/tree/main/src/main/java/com/dynatrace/hash4j/distinctcount"
    click C "https://github.com/dynatrace-oss/hash4j/tree/main/src/main/java/com/dynatrace/hash4j/similarity"
    click D "https://github.com/dynatrace-oss/hash4j/blob/main/src/main/java/com/dynatrace/hash4j/file/Imohash1_0_2.java"
    click E "https://github.com/dynatrace-oss/hash4j/tree/main/src/main/java/com/dynatrace/hash4j/consistent"
    click F "https://github.com/dynatrace-oss/hash4j/tree/main/src/main/java/com/dynatrace/hash4j/random"
    click G "https://github.com/dynatrace-oss/hash4j/tree/main/src/main/java/com/dynatrace/hash4j/util"
    click T "https://github.com/dynatrace-oss/hash4j/tree/main/src/test"
    click BM "https://github.com/dynatrace-oss/hash4j/tree/main/src/jmh"
    click X "https://github.com/dynatrace-oss/hash4j/tree/main/reference-implementations"

    classDef module fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef aux fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef core fill:#fff3cd,stroke:#856404,stroke-width:2px;
```
