# fireindark707/python-schema-matching architecture

> Architecture diagram of the GitHub repository fireindark707/python-schema-matching, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fireindark707/python-schema-matching
- Repository: https://github.com/fireindark707/python-schema-matching

## Overview

##### Project Type and Purpose
- This is an open-source Python library and CLI tool (not a web or full-stack app) for schema matching between two tables (CSV/JSON) using a hybrid ML pipeline.
- It provides both a Python API (`schema_matching` function) and command-line entrypoints (`relation_features.py`, `train.py`, `cal_column_similarity.py`).

##### File Structure Analysis
- Top-level scripts:
- `relation_features.py`, `self_features.py` – extract column-level and instance-level features
- `train.py` – orchestrates XGBoost model training on feature sets
- `cal_column_similarity.py` – loads trained models, runs inference, outputs similarity matrices & predicted pairs
- `__init__.py` – defines the `schema_matching` API function
- `utils.py` – shared utilities (file I/O, threshold computation, strategy logic)
- Directories:
- `Training Data/` & `Test Data/` – sample data folders with `Table1.csv`, `Table2.csv`, optional `mapping.txt`
- `model/` – versioned folders containing multiple `.model` and `.threshold` files (one per feature subset or fold)
- Config & dependencies: `requirements.txt`, license, citation file, README.

##### Architecture from README
- Two main workflows:
A. **Training Pipeline**
1. Ingest paired tables + mapping.txt
2. Feature construction (`relation_features.py`, `self_features.py`)
3. Train one or more XGBoost models (`train.py`)
4. Write model artifacts + thresholds to `model/yyyy-mm-dd-hh-mm-ss/`
B. **Inference Pipeline**
1. Ingest Table1/Table2 (csv/json/jsonl) via CLI or API
2. Reuse feature constructors
3. Load ensemble of XGBoost models + thresholds
4. Compute raw similarity scores, average across models
5. Apply threshold & matching strategy
6. Output `similarity_matrix_value.csv`, `similarity_matrix_label.csv`, and predicted pairs

a. Main Components
- User Entrypoints:
- Python API (`schema_matching()`)
- CLI scripts (`cal_column_similarity.py`, `train.py`)
- Data Sources:
- Training Data folders (`Table1.csv`, `Table2.csv`, `mapping.txt`)
- Test Data folders (`Table1.csv`/`.json`, `Table2.csv`/`.json`)
- Feature Engineering Module:
- Column–level features (`relation_features.py`)
- Instance–level features (`self_features.py`)
- Model Training Module:
- XGBoost training engine (`train.py`)
- Model Storage:
- Versioned model directories containing `.model` and `.threshold`
- Inference Module:
- Scoring engine (`cal_column_similarity.py`)
- Strategy logic (one-to-one, one-to-many, many-to-many) in `utils.py`
- Output:
- CSVs for similarity values and labels
- Of matched column-pairs

b. Relationships & Data Flow
- User CLI/API → Data Loader → Feature Constructors → (Training: → XGBoost Trainer → Model Store)
- Inference: Feature Constructors + Model Loader → Ensemble Scorer → Threshold & Strategy → Outputs

c. Architectural Patterns & Principles
- Pipeline pattern: separate stages for data ingestion, feature extraction, model training/inference.
- Ensemble pattern: multiple XGBoost models with averaging.
- Separation of concerns: clear boundaries between feature logic, model logic, I/O, and strategy.
- Extensibility: ability to plug in new features or models by adding scripts and updating model folder.

d. Key Technologies & Libraries
- pandas, numpy for data manipulation
- nltk & python-dateutil for text/date features
- sentence-transformers for multilingual text embeddings (TSM cosine)
- xgboost for classification models
- strsimpy for string similarity metrics

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        TD1["Training Data"]:::data
        click TD1 "https://github.com/fireindark707/python-schema-matching/tree/main/Training Data/"
        TD2["Test Data"]:::data
        click TD2 "https://github.com/fireindark707/python-schema-matching/tree/main/Test Data/"
    end

    %% Entrypoints
    subgraph "Entrypoints"
        direction TB
        API["Python API: schema_matching()"]:::api
        click API "https://github.com/fireindark707/python-schema-matching/blob/main/__init__.py"
        CLI_feat_rel["CLI: relation_features.py"]:::process
        click CLI_feat_rel "https://github.com/fireindark707/python-schema-matching/blob/main/relation_features.py"
        CLI_feat_self["CLI: self_features.py"]:::process
        click CLI_feat_self "https://github.com/fireindark707/python-schema-matching/blob/main/self_features.py"
        CLI_train["CLI: train.py"]:::process
        click CLI_train "https://github.com/fireindark707/python-schema-matching/blob/main/train.py"
        CLI_infer["CLI: cal_column_similarity.py"]:::process
        click CLI_infer "https://github.com/fireindark707/python-schema-matching/blob/main/cal_column_similarity.py"
    end

    %% Core Modules
    subgraph "Core Modules"
        direction TB
        Utils["Utils & Strategy Logic"]:::util
        click Utils "https://github.com/fireindark707/python-schema-matching/blob/main/utils.py"
        subgraph "Feature Engineering"
            direction TB
            RelFeat["Column-level Features"]:::process
            click RelFeat "https://github.com/fireindark707/python-schema-matching/blob/main/relation_features.py"
            InstFeat["Instance-level Features"]:::process
            click InstFeat "https://github.com/fireindark707/python-schema-matching/blob/main/self_features.py"
        end
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        TD1 --> RelFeat
        TD1 --> InstFeat
        RelFeat --> TrainProc
        InstFeat --> TrainProc
        TrainProc["XGBoost Trainer"]:::process
        click CLI_train "https://github.com/fireindark707/python-schema-matching/blob/main/train.py"
        TrainProc --> Models
    end

    %% Model Storage
    subgraph "Model Storage"
        direction TB
        Models["Versioned Model Dir"]:::storage
        click Models "https://github.com/fireindark707/python-schema-matching/tree/main/model/2022-04-12-12-06-32/"
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        TD2 --> RelFeat
        TD2 --> InstFeat
        RelFeat --> InferProc
        InstFeat --> InferProc
        InferProc["Scoring Engine"]:::process
        click CLI_infer "https://github.com/fireindark707/python-schema-matching/blob/main/cal_column_similarity.py"
        Models --> InferProc
        Utils --> InferProc
        InferProc --> Output
    end

    %% Outputs
    Output["Outputs:\nsimilarity_matrix_value.csv,\nsimilarity_matrix_label.csv,\npredicted_pairs"]:::output

    %% API Flow
    API --> RelFeat
    API --> InstFeat
    API --> Models
    API --> Output

    %% Config & Metadata
    subgraph "Config & Metadata"
        direction TB
        Req["requirements.txt"]:::config
        click Req "https://github.com/fireindark707/python-schema-matching/blob/main/requirements.txt"
        CIT["CITATION.cff"]:::config
        click CIT "https://github.com/fireindark707/python-schema-matching/blob/main/CITATION.cff"
        LIC["LICENSE"]:::config
        click LIC "https://github.com/fireindark707/python-schema-matching/tree/main/LICENSE"
        RD["README.md"]:::config
        click RD "https://github.com/fireindark707/python-schema-matching/blob/main/README.md"
    end

    %% Styles
    classDef data fill:#D0E8FF,stroke:#0366D6,stroke-width:1px;
    classDef process fill:#FFF4D1,stroke:#D4A017,stroke-width:1px;
    classDef util fill:#FFEFD5,stroke:#E9967A,stroke-width:1px;
    classDef storage fill:#E0FFE0,stroke:#228B22,stroke-width:1px;
    classDef output fill:#F0F0F0,stroke:#666666,stroke-width:1px;
    classDef api fill:#FFF0F5,stroke:#C71585,stroke-width:1px;
    classDef config fill:#F5F5F5,stroke:#8B008B,stroke-width:1px;
```
