# graphlet-ai/serf architecture

> Architecture diagram of the GitHub repository graphlet-ai/serf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/graphlet-ai/serf
- Repository: https://github.com/graphlet-ai/serf

## Overview

SERF resolves heterogeneous records through semantic blocking, whole-block LLM partition matching, canonical merging, and repeated rounds that give previously separated records another chance to meet. The CLI exposes the framework; schemas and field semantics govern canonicalization, while evaluation and DSPy optimization support measured prompt improvement. A Spark implementation provides an optional distributed path. Edges follow explicit README workflow statements or references in the SOURCE INDEX; unsampled modules are represented without guessing their internal call relationships.

Main entry points: [`src/serf/match/matcher.py`](https://github.com/graphlet-ai/serf/blob/main/src/serf/match/matcher.py) (Block matcher), [`src/serf/pipeline.py`](https://github.com/graphlet-ai/serf/blob/main/src/serf/pipeline.py) (Iterative ER pipeline), [`src/serf/cli/main.py`](https://github.com/graphlet-ai/serf/blob/main/src/serf/cli/main.py) (CLI commands), [`src/serf/block/pipeline.py`](https://github.com/graphlet-ai/serf/blob/main/src/serf/block/pipeline.py) (Semantic blocking)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and orchestration"]
  node_cli["CLI commands<br/>[main.py]"]
  node_pipeline["Iterative ER pipeline<br/>[pipeline.py]"]
  node_config["Configuration<br/>[config.py]"]
end

subgraph group_resolution["Resolution workflow"]
  node_blocking["Semantic blocking<br/>[pipeline.py]"]
  node_embed["Embedding subprocess"]
  node_faiss["FAISS blocking<br/>[faiss_blocker.py]"]
  node_matcher["Block matcher<br/>[matcher.py]"]
  node_matchrun["Match orchestration<br/>[run.py]"]
  node_partition["Entity partitioning<br/>[partition.py]"]
  node_canonical["Canonical merging<br/>[canonical.py]"]
  node_semantics["Field merge semantics<br/>[semantics.py]"]
  node_lineage["Lineage conservation<br/>[conservation.py]"]
end

subgraph group_llm["LLM and schemas"]
  node_dspy["DSPy signatures<br/>[signatures.py]"]
  node_adapter["XML response adapter<br/>[adapter.py]"]
  node_lm["LLM factory<br/>[lm.py]"]
  node_dsignatures["Dataset signatures"]
  node_schema["Schema policies<br/>[spec.py]"]
  node_schema_gen["Schema generation<br/>[generate.py]"]
end

subgraph group_quality["Evaluation and tuning"]
  node_eval["Evaluation metrics<br/>[metrics.py]"]
  node_benchmarks["Benchmark datasets<br/>[benchmarks.py]"]
  node_splits["Train and holdout splits<br/>[splits.py]"]
  node_optimizer["GEPA optimization<br/>[optimize.py]"]
end

subgraph group_distributed["Distributed processing"]
  node_spark["Spark pipeline<br/>[pipeline.py]"]
  node_sparkgraph["Graph operations<br/>[graph.py]"]
  node_iceberg["Iceberg I/O<br/>[iceberg.py]"]
end

node_user(("Framework user"))

node_user -->|"invokes"| node_cli
node_user -->|"runs"| node_pipeline
node_cli -->|"routes commands"| node_pipeline
node_pipeline -->|"blocks records"| node_blocking
node_blocking -->|"embeds via"| node_embed
node_blocking -->|"clusters with"| node_faiss
node_pipeline -->|"matches blocks"| node_matcher
node_pipeline -->|"merges entities"| node_canonical
node_pipeline -->|"re-blocks rounds"| node_blocking
node_matchrun -->|"runs"| node_matcher
node_matcher -->|"predicts with"| node_dspy
node_matcher -->|"parses through"| node_adapter
node_matcher -->|"creates model"| node_lm
node_matchrun -->|"uses"| node_partition
node_canonical -->|"applies policies"| node_semantics
node_cli -->|"exposes schema tools"| node_schema_gen
node_schema_gen -->|"reads specification"| node_schema
node_schema -->|"defines policies"| node_semantics
node_cli -->|"runs benchmark workflows"| node_benchmarks
node_benchmarks -->|"provides evaluation data"| node_eval
node_splits -->|"uses benchmark config"| node_benchmarks
node_optimizer -->|"optimizes matching"| node_matcher
node_optimizer -->|"scores trials"| node_eval
node_optimizer -->|"uses holdouts"| node_splits
node_matchrun -->|"merges matched entities"| node_canonical
node_spark -->|"uses embedding subprocess"| node_embed
node_spark -->|"uses matching"| node_matcher
node_pipeline -->|"reads settings"| node_config
node_matcher -.->|"supports dataset modes"| node_dsignatures

click node_cli "https://github.com/graphlet-ai/serf/blob/main/src/serf/cli/main.py"
click node_pipeline "https://github.com/graphlet-ai/serf/blob/main/src/serf/pipeline.py"
click node_config "https://github.com/graphlet-ai/serf/blob/main/src/serf/config.py"
click node_blocking "https://github.com/graphlet-ai/serf/blob/main/src/serf/block/pipeline.py"
click node_embed "https://github.com/graphlet-ai/serf/blob/main/src/serf/block/subprocess_embed.py"
click node_faiss "https://github.com/graphlet-ai/serf/blob/main/src/serf/block/faiss_blocker.py"
click node_matcher "https://github.com/graphlet-ai/serf/blob/main/src/serf/match/matcher.py"
click node_matchrun "https://github.com/graphlet-ai/serf/blob/main/src/serf/match/run.py"
click node_partition "https://github.com/graphlet-ai/serf/blob/main/src/serf/match/partition.py"
click node_canonical "https://github.com/graphlet-ai/serf/blob/main/src/serf/merge/canonical.py"
click node_semantics "https://github.com/graphlet-ai/serf/blob/main/src/serf/merge/semantics.py"
click node_lineage "https://github.com/graphlet-ai/serf/blob/main/src/serf/merge/conservation.py"
click node_dspy "https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/signatures.py"
click node_adapter "https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/adapter.py"
click node_lm "https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/lm.py"
click node_dsignatures "https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/dataset_signatures.py"
click node_schema "https://github.com/graphlet-ai/serf/blob/main/src/serf/schema/spec.py"
click node_schema_gen "https://github.com/graphlet-ai/serf/blob/main/src/serf/schema/generate.py"
click node_eval "https://github.com/graphlet-ai/serf/blob/main/src/serf/eval/metrics.py"
click node_benchmarks "https://github.com/graphlet-ai/serf/blob/main/src/serf/eval/benchmarks.py"
click node_splits "https://github.com/graphlet-ai/serf/blob/main/src/serf/eval/splits.py"
click node_optimizer "https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/optimize.py"
click node_spark "https://github.com/graphlet-ai/serf/blob/main/src/serf/spark/pipeline.py"
click node_sparkgraph "https://github.com/graphlet-ai/serf/blob/main/src/serf/spark/graph.py"
click node_iceberg "https://github.com/graphlet-ai/serf/blob/main/src/serf/spark/iceberg.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_pipeline,node_config,node_user toneBlue
class node_blocking,node_embed,node_faiss,node_matcher,node_matchrun,node_partition,node_canonical,node_semantics,node_lineage toneAmber
class node_dspy,node_adapter,node_lm,node_dsignatures,node_schema,node_schema_gen toneMint
class node_eval,node_benchmarks,node_splits,node_optimizer toneRose
class node_spark,node_sparkgraph,node_iceberg toneIndigo
```

## Components

### Entry and orchestration

- [CLI commands](https://github.com/graphlet-ai/serf/blob/main/src/serf/cli/main.py): `src/serf/cli/main.py`
- [Iterative ER pipeline](https://github.com/graphlet-ai/serf/blob/main/src/serf/pipeline.py): `src/serf/pipeline.py`
- [Configuration](https://github.com/graphlet-ai/serf/blob/main/src/serf/config.py): `src/serf/config.py`

### Resolution workflow

- [Semantic blocking](https://github.com/graphlet-ai/serf/blob/main/src/serf/block/pipeline.py): `src/serf/block/pipeline.py`
- [Embedding subprocess](https://github.com/graphlet-ai/serf/blob/main/src/serf/block/subprocess_embed.py): `src/serf/block/subprocess_embed.py`
- [FAISS blocking](https://github.com/graphlet-ai/serf/blob/main/src/serf/block/faiss_blocker.py): `src/serf/block/faiss_blocker.py`
- [Block matcher](https://github.com/graphlet-ai/serf/blob/main/src/serf/match/matcher.py): `src/serf/match/matcher.py`
- [Match orchestration](https://github.com/graphlet-ai/serf/blob/main/src/serf/match/run.py): `src/serf/match/run.py`
- [Entity partitioning](https://github.com/graphlet-ai/serf/blob/main/src/serf/match/partition.py): `src/serf/match/partition.py`
- [Canonical merging](https://github.com/graphlet-ai/serf/blob/main/src/serf/merge/canonical.py): `src/serf/merge/canonical.py`
- [Field merge semantics](https://github.com/graphlet-ai/serf/blob/main/src/serf/merge/semantics.py): `src/serf/merge/semantics.py`
- [Lineage conservation](https://github.com/graphlet-ai/serf/blob/main/src/serf/merge/conservation.py): `src/serf/merge/conservation.py`

### LLM and schemas

- [DSPy signatures](https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/signatures.py): `src/serf/dspy/signatures.py`
- [XML response adapter](https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/adapter.py): `src/serf/dspy/adapter.py`
- [LLM factory](https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/lm.py): `src/serf/dspy/lm.py`
- [Dataset signatures](https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/dataset_signatures.py): `src/serf/dspy/dataset_signatures.py`
- [Schema policies](https://github.com/graphlet-ai/serf/blob/main/src/serf/schema/spec.py): `src/serf/schema/spec.py`
- [Schema generation](https://github.com/graphlet-ai/serf/blob/main/src/serf/schema/generate.py): `src/serf/schema/generate.py`

### Evaluation and tuning

- [Evaluation metrics](https://github.com/graphlet-ai/serf/blob/main/src/serf/eval/metrics.py): `src/serf/eval/metrics.py`
- [Benchmark datasets](https://github.com/graphlet-ai/serf/blob/main/src/serf/eval/benchmarks.py): `src/serf/eval/benchmarks.py`
- [Train and holdout splits](https://github.com/graphlet-ai/serf/blob/main/src/serf/eval/splits.py): `src/serf/eval/splits.py`
- [GEPA optimization](https://github.com/graphlet-ai/serf/blob/main/src/serf/dspy/optimize.py): `src/serf/dspy/optimize.py`

### Distributed processing

- [Spark pipeline](https://github.com/graphlet-ai/serf/blob/main/src/serf/spark/pipeline.py): `src/serf/spark/pipeline.py`
- [Graph operations](https://github.com/graphlet-ai/serf/blob/main/src/serf/spark/graph.py): `src/serf/spark/graph.py`
- [Iceberg I/O](https://github.com/graphlet-ai/serf/blob/main/src/serf/spark/iceberg.py): `src/serf/spark/iceberg.py`

### Other components

- **Framework user**

## Connections

- Framework user → CLI commands: invokes
- Framework user → Iterative ER pipeline: runs
- CLI commands → Iterative ER pipeline: routes commands (evidence: `src/serf/cli/main.py`)
- Iterative ER pipeline → Semantic blocking: blocks records
- Semantic blocking → Embedding subprocess: embeds via (evidence: `src/serf/block/pipeline.py`)
- Semantic blocking → FAISS blocking: clusters with
- Iterative ER pipeline → Block matcher: matches blocks (evidence: `src/serf/pipeline.py`)
- Iterative ER pipeline → Canonical merging: merges entities
- Iterative ER pipeline → Semantic blocking: re-blocks rounds
- Match orchestration → Block matcher: runs (evidence: `src/serf/match/run.py`)
- Block matcher → DSPy signatures: predicts with (evidence: `src/serf/match/matcher.py`)
- Block matcher → XML response adapter: parses through (evidence: `src/serf/match/matcher.py`)
- Block matcher → LLM factory: creates model (evidence: `src/serf/match/matcher.py`)
- Match orchestration → Entity partitioning: uses (evidence: `src/serf/match/run.py`)
- Canonical merging → Field merge semantics: applies policies (evidence: `src/serf/merge/canonical.py`)
- CLI commands → Schema generation: exposes schema tools (evidence: `src/serf/cli/main.py`)
- Schema generation → Schema policies: reads specification (evidence: `src/serf/schema/generate.py`)
- Schema policies → Field merge semantics: defines policies (evidence: `src/serf/schema/spec.py`)
- CLI commands → Benchmark datasets: runs benchmark workflows (evidence: `src/serf/cli/main.py`)
- Benchmark datasets → Evaluation metrics: provides evaluation data (evidence: `src/serf/eval/benchmarks.py`)
- Train and holdout splits → Benchmark datasets: uses benchmark config (evidence: `src/serf/eval/splits.py`)
- GEPA optimization → Block matcher: optimizes matching (evidence: `src/serf/dspy/optimize.py`)
- GEPA optimization → Evaluation metrics: scores trials (evidence: `src/serf/dspy/optimize.py`)
- GEPA optimization → Train and holdout splits: uses holdouts (evidence: `src/serf/dspy/optimize.py`)
- Match orchestration → Canonical merging: merges matched entities (evidence: `src/serf/match/run.py`)
- Spark pipeline → Embedding subprocess: uses embedding subprocess (evidence: `src/serf/spark/pipeline.py`)
- Spark pipeline → Block matcher: uses matching (evidence: `src/serf/spark/pipeline.py`)
- Iterative ER pipeline → Configuration: reads settings (evidence: `src/serf/pipeline.py`)
- Block matcher → Dataset signatures: supports dataset modes (evidence: `src/serf/match/run.py`)
