# zinggai/zingg architecture

> Architecture diagram of the GitHub repository zinggai/zingg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zinggai/zingg
- Repository: https://github.com/zinggai/zingg

## Overview

Zingg is a Spark-based entity-resolution and master-data tool. Its principal workflow configures input data and field comparisons, builds training examples through active labeling, learns blocking and similarity models, then matches records and emits linked or clustered results. The README also documents broad file, database, and cloud connectors, domain-specific matching functions, and stop-word recommendations. The map uses the supplied tree and excerpts; several phase implementations and connector details were not sampled, so uncertain runtime wiring is omitted.

Main entry points: [`common/core/src/main/java/zingg/common/core/executor/Matcher.java`](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Matcher.java) (Pair matching), [`common/client/src/main/java/zingg/common/client/Client.java`](https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/Client.java) (Client runner), [`spark/client/src/main/java/zingg/spark/client/util/SparkPipeUtil.java`](https://github.com/zinggai/zingg/blob/main/spark/client/src/main/java/zingg/spark/client/util/SparkPipeUtil.java) (Spark pipe I/O), [`spark/client/src/main/java/zingg/spark/client/SparkClient.java`](https://github.com/zinggai/zingg/blob/main/spark/client/src/main/java/zingg/spark/client/SparkClient.java) (Spark client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Client API"]
  node_pyapi["Python API<br/>[client.py]"]
  node_client["Client runner<br/>[Client.java]"]
  node_sparkclient["Spark client<br/>[SparkClient.java]"]
  node_config["Arguments"]
end

subgraph group_data["Data Access"]
  node_pipes["Pipe abstraction<br/>[Pipe.java]"]
  node_sparkpipe["Spark pipe I/O<br/>[SparkPipeUtil.java]"]
  node_context["Spark context"]
end

subgraph group_learning["Learning"]
  node_preprocess["Preprocessing"]
  node_training["Training examples"]
  node_labeling["Interactive labeling<br/>[Labeller.java]"]
  node_trainer["Model training<br/>[Trainer.java]"]
  node_model["Similarity model<br/>[Model.java]"]
end

subgraph group_resolution["Resolution"]
  node_blocking["Auto blocking"]
  node_matcher["Pair matching<br/>[Matcher.java]"]
  node_similarity["Similarity functions"]
  node_linker["Entity linking<br/>[Linker.java]"]
  node_outputs["Match outputs"]
  node_stopwords["Stop-word advice"]
end

node_user(("Engineer"))

node_user -->|"configures and runs"| node_pyapi
node_user -->|"starts phase"| node_client
node_pyapi -->|"delegates"| node_sparkclient
node_client -->|"loads arguments"| node_config
node_client -->|"creates client"| node_sparkclient
node_sparkclient -->|"initializes runtime"| node_context
node_config -->|"defines data pipes"| node_pipes
node_sparkpipe -->|"implements I/O"| node_pipes
node_context -->|"sets pipe utility"| node_sparkpipe
node_training -->|"prepares records"| node_preprocess
node_training -->|"supplies candidates"| node_labeling
node_labeling -->|"provides labels"| node_trainer
node_trainer -->|"trains"| node_model
node_blocking -->|"groups candidates"| node_matcher
node_matcher -->|"predicts with"| node_model
node_matcher -->|"scores pairs"| node_similarity
node_matcher -->|"passes matches"| node_linker
node_linker -->|"builds results"| node_outputs
node_sparkpipe -->|"writes results"| node_outputs
node_stopwords -.->|"recommends terms"| node_user

click node_pyapi "https://github.com/zinggai/zingg/blob/main/python/zingg/client.py"
click node_client "https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/Client.java"
click node_sparkclient "https://github.com/zinggai/zingg/blob/main/spark/client/src/main/java/zingg/spark/client/SparkClient.java"
click node_config "https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/arguments/ArgumentServiceImpl.java"
click node_pipes "https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/pipe/Pipe.java"
click node_sparkpipe "https://github.com/zinggai/zingg/blob/main/spark/client/src/main/java/zingg/spark/client/util/SparkPipeUtil.java"
click node_context "https://github.com/zinggai/zingg/blob/main/spark/core/src/main/java/zingg/spark/core/context/ZinggSparkContext.java"
click node_preprocess "https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/preprocess"
click node_training "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/TrainingDataFinder.java"
click node_labeling "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Labeller.java"
click node_trainer "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Trainer.java"
click node_model "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/model/Model.java"
click node_blocking "https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/block"
click node_matcher "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Matcher.java"
click node_similarity "https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/similarity/function"
click node_linker "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Linker.java"
click node_outputs "https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/match/output"
click node_stopwords "https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/recommender/StopWordsRecommender.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pyapi,node_client,node_sparkclient,node_config toneBlue
class node_pipes,node_sparkpipe,node_context toneAmber
class node_preprocess,node_training,node_labeling,node_trainer,node_model toneMint
class node_blocking,node_matcher,node_similarity,node_linker,node_outputs,node_stopwords toneRose
class node_user toneIndigo
```

## Components

### Client API

- [Python API](https://github.com/zinggai/zingg/blob/main/python/zingg/client.py): `python/zingg/client.py`
- [Client runner](https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/Client.java): `common/client/src/main/java/zingg/common/client/Client.java`
- [Spark client](https://github.com/zinggai/zingg/blob/main/spark/client/src/main/java/zingg/spark/client/SparkClient.java): `spark/client/src/main/java/zingg/spark/client/SparkClient.java`
- [Arguments](https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/arguments/ArgumentServiceImpl.java): `common/client/src/main/java/zingg/common/client/arguments/ArgumentServiceImpl.java`

### Data Access

- [Pipe abstraction](https://github.com/zinggai/zingg/blob/main/common/client/src/main/java/zingg/common/client/pipe/Pipe.java): `common/client/src/main/java/zingg/common/client/pipe/Pipe.java`
- [Spark pipe I/O](https://github.com/zinggai/zingg/blob/main/spark/client/src/main/java/zingg/spark/client/util/SparkPipeUtil.java): `spark/client/src/main/java/zingg/spark/client/util/SparkPipeUtil.java`
- [Spark context](https://github.com/zinggai/zingg/blob/main/spark/core/src/main/java/zingg/spark/core/context/ZinggSparkContext.java): `spark/core/src/main/java/zingg/spark/core/context/ZinggSparkContext.java`

### Learning

- [Preprocessing](https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/preprocess): `common/core/src/main/java/zingg/common/core/preprocess`
- [Training examples](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/TrainingDataFinder.java): `common/core/src/main/java/zingg/common/core/executor/TrainingDataFinder.java`
- [Interactive labeling](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Labeller.java): `common/core/src/main/java/zingg/common/core/executor/Labeller.java`
- [Model training](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Trainer.java): `common/core/src/main/java/zingg/common/core/executor/Trainer.java`
- [Similarity model](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/model/Model.java): `common/core/src/main/java/zingg/common/core/model/Model.java`

### Resolution

- [Auto blocking](https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/block): `common/core/src/main/java/zingg/common/core/block`
- [Pair matching](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Matcher.java): `common/core/src/main/java/zingg/common/core/executor/Matcher.java`
- [Similarity functions](https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/similarity/function): `common/core/src/main/java/zingg/common/core/similarity/function`
- [Entity linking](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/executor/Linker.java): `common/core/src/main/java/zingg/common/core/executor/Linker.java`
- [Match outputs](https://github.com/zinggai/zingg/tree/main/common/core/src/main/java/zingg/common/core/match/output): `common/core/src/main/java/zingg/common/core/match/output`
- [Stop-word advice](https://github.com/zinggai/zingg/blob/main/common/core/src/main/java/zingg/common/core/recommender/StopWordsRecommender.java): `common/core/src/main/java/zingg/common/core/recommender/StopWordsRecommender.java`

### Other components

- **Engineer**

## Connections

- Engineer → Python API: configures and runs
- Engineer → Client runner: starts phase
- Python API → Spark client: delegates
- Client runner → Arguments: loads arguments
- Client runner → Spark client: creates client
- Spark client → Spark context: initializes runtime
- Arguments → Pipe abstraction: defines data pipes
- Spark pipe I/O → Pipe abstraction: implements I/O
- Spark context → Spark pipe I/O: sets pipe utility
- Training examples → Preprocessing: prepares records
- Training examples → Interactive labeling: supplies candidates
- Interactive labeling → Model training: provides labels
- Model training → Similarity model: trains
- Auto blocking → Pair matching: groups candidates
- Pair matching → Similarity model: predicts with
- Pair matching → Similarity functions: scores pairs
- Pair matching → Entity linking: passes matches
- Entity linking → Match outputs: builds results
- Spark pipe I/O → Match outputs: writes results
- Stop-word advice → Engineer: recommends terms