How zinggai/zingg works
Zingg is a Spark-based entity-resolution and master-data tool. Its principal workflow configures input data and field comparisons, builds training examples through active labeling, learns blocking and similarity models, then matches records and emits linked or clustered results. The README also documents broad file, database, and cloud connectors, domain-specific matching functions, and stop-word recommendations. The map uses the supplied tree and excerpts; several phase implementations and connector details were not sampled, so uncertain runtime wiring is omitted.
Main entry points: Matcher.java Pair matching · Client.java Client runner · SparkPipeUtil.java Spark pipe I/O · SparkClient.java Spark client
Components 19
Client API
- Python API
python/zingg/client.py - Client runner
common/client/src/main/java/zingg/common/client/Client.java - Spark client
spark/client/src/main/java/zingg/spark/client/SparkClient.java - Arguments
common/client/src/main/java/zingg/common/client/arguments/ArgumentServiceImpl.java
Data Access
- Pipe abstraction
common/client/src/main/java/zingg/common/client/pipe/Pipe.java - Spark pipe I/O
spark/client/src/main/java/zingg/spark/client/util/SparkPipeUtil.java - Spark context
spark/core/src/main/java/zingg/spark/core/context/ZinggSparkContext.java
Learning
- Preprocessing
common/core/src/main/java/zingg/common/core/preprocess - Training examples
common/core/src/main/java/zingg/common/core/executor/TrainingDataFinder.java - Interactive labeling
common/core/src/main/java/zingg/common/core/executor/Labeller.java - Model training
common/core/src/main/java/zingg/common/core/executor/Trainer.java - Similarity model
common/core/src/main/java/zingg/common/core/model/Model.java
Resolution
- Auto blocking
common/core/src/main/java/zingg/common/core/block - Pair matching
common/core/src/main/java/zingg/common/core/executor/Matcher.java - Similarity functions
common/core/src/main/java/zingg/common/core/similarity/function - Entity linking
common/core/src/main/java/zingg/common/core/executor/Linker.java - Match outputs
common/core/src/main/java/zingg/common/core/match/output - Stop-word advice
common/core/src/main/java/zingg/common/core/recommender/StopWordsRecommender.java
Other components
- Engineer
Connections 20
- Engineer → Python API: configures and runs
- Engineer → Client runner: starts phase
- Python API → Spark client: delegates
- Client runner → Arguments: loads arguments
- Client runner → Spark client: creates client
- Spark client → Spark context: initializes runtime
- Arguments → Pipe abstraction: defines data pipes
- Spark pipe I/O → Pipe abstraction: implements I/O
- Spark context → Spark pipe I/O: sets pipe utility
- Training examples → Preprocessing: prepares records
- Training examples → Interactive labeling: supplies candidates
- Interactive labeling → Model training: provides labels
- Model training → Similarity model: trains
- Auto blocking → Pair matching: groups candidates
- Pair matching → Similarity model: predicts with
- Pair matching → Similarity functions: scores pairs
- Pair matching → Entity linking: passes matches
- Entity linking → Match outputs: builds results
- Spark pipe I/O → Match outputs: writes results
- Stop-word advice → Engineer: recommends terms