How graphlet-ai/serf works
SERF resolves heterogeneous records through semantic blocking, whole-block LLM partition matching, canonical merging, and repeated rounds that give previously separated records another chance to meet. The CLI exposes the framework; schemas and field semantics govern canonicalization, while evaluation and DSPy optimization support measured prompt improvement. A Spark implementation provides an optional distributed path. Edges follow explicit README workflow statements or references in the SOURCE INDEX; unsampled modules are represented without guessing their internal call relationships.
Main entry points: matcher.py Block matcher · serf/pipeline.py Iterative ER pipeline · main.py CLI commands · block/pipeline.py Semantic blocking
Components 26
Entry and orchestration
- CLI commands
src/serf/cli/main.py - Iterative ER pipeline
src/serf/pipeline.py - Configuration
src/serf/config.py
Resolution workflow
- Semantic blocking
src/serf/block/pipeline.py - Embedding subprocess
src/serf/block/subprocess_embed.py - FAISS blocking
src/serf/block/faiss_blocker.py - Block matcher
src/serf/match/matcher.py - Match orchestration
src/serf/match/run.py - Entity partitioning
src/serf/match/partition.py - Canonical merging
src/serf/merge/canonical.py - Field merge semantics
src/serf/merge/semantics.py - Lineage conservation
src/serf/merge/conservation.py
LLM and schemas
- DSPy signatures
src/serf/dspy/signatures.py - XML response adapter
src/serf/dspy/adapter.py - LLM factory
src/serf/dspy/lm.py - Dataset signatures
src/serf/dspy/dataset_signatures.py - Schema policies
src/serf/schema/spec.py - Schema generation
src/serf/schema/generate.py
Evaluation and tuning
- Evaluation metrics
src/serf/eval/metrics.py - Benchmark datasets
src/serf/eval/benchmarks.py - Train and holdout splits
src/serf/eval/splits.py - GEPA optimization
src/serf/dspy/optimize.py
Distributed processing
- Spark pipeline
src/serf/spark/pipeline.py - Graph operations
src/serf/spark/graph.py - Iceberg I/O
src/serf/spark/iceberg.py
Other components
- Framework user
Connections 29
- Framework user → CLI commands: invokes
- Framework user → Iterative ER pipeline: runs
- CLI commands → Iterative ER pipeline: routes commands
src/serf/cli/main.py - Iterative ER pipeline → Semantic blocking: blocks records
- Semantic blocking → Embedding subprocess: embeds via
src/serf/block/pipeline.py - Semantic blocking → FAISS blocking: clusters with
- Iterative ER pipeline → Block matcher: matches blocks
src/serf/pipeline.py - Iterative ER pipeline → Canonical merging: merges entities
- Iterative ER pipeline → Semantic blocking: re-blocks rounds
- Match orchestration → Block matcher: runs
src/serf/match/run.py - Block matcher → DSPy signatures: predicts with
src/serf/match/matcher.py - Block matcher → XML response adapter: parses through
src/serf/match/matcher.py - Block matcher → LLM factory: creates model
src/serf/match/matcher.py - Match orchestration → Entity partitioning: uses
src/serf/match/run.py - Canonical merging → Field merge semantics: applies policies
src/serf/merge/canonical.py - CLI commands → Schema generation: exposes schema tools
src/serf/cli/main.py - Schema generation → Schema policies: reads specification
src/serf/schema/generate.py - Schema policies → Field merge semantics: defines policies
src/serf/schema/spec.py - CLI commands → Benchmark datasets: runs benchmark workflows
src/serf/cli/main.py - Benchmark datasets → Evaluation metrics: provides evaluation data
src/serf/eval/benchmarks.py - Train and holdout splits → Benchmark datasets: uses benchmark config
src/serf/eval/splits.py - GEPA optimization → Block matcher: optimizes matching
src/serf/dspy/optimize.py - GEPA optimization → Evaluation metrics: scores trials
src/serf/dspy/optimize.py - GEPA optimization → Train and holdout splits: uses holdouts
src/serf/dspy/optimize.py - Match orchestration → Canonical merging: merges matched entities
src/serf/match/run.py - Spark pipeline → Embedding subprocess: uses embedding subprocess
src/serf/spark/pipeline.py - Spark pipeline → Block matcher: uses matching
src/serf/spark/pipeline.py - Iterative ER pipeline → Configuration: reads settings
src/serf/pipeline.py - Block matcher → Dataset signatures: supports dataset modes
src/serf/match/run.py