How fireindark707/python-schema-matching works
Project Type and Purpose
• This is an open-source Python library and CLI tool (not a web or full-stack app) for schema matching between two tables (CSV/JSON) using a hybrid ML pipeline.
• It provides both a Python API (schema_matching function) and command-line entrypoints (relation_features.py, train.py, cal_column_similarity.py).
File Structure Analysis
• Top-level scripts:
• relation_features.py, self_features.py – extract column-level and instance-level features
• train.py – orchestrates XGBoost model training on feature sets
• cal_column_similarity.py – loads trained models, runs inference, outputs similarity matrices & predicted pairs
• __init__.py – defines the schema_matching API function
• utils.py – shared utilities (file I/O, threshold computation, strategy logic)
• Directories:
• Training Data/ & Test Data/ – sample data folders with Table1.csv, Table2.csv, optional mapping.txt
• model/ – versioned folders containing multiple .model and .threshold files (one per feature subset or fold)
• Config & dependencies: requirements.txt, license, citation file, README.
Architecture from README
• Two main workflows:
A. Training Pipeline
1. Ingest paired tables + mapping.txt
2. Feature construction (relation_features.py, self_features.py)
3. Train one or more XGBoost models (train.py)
4. Write model artifacts + thresholds to model/yyyy-mm-dd-hh-mm-ss/
B. Inference Pipeline
1. Ingest Table1/Table2 (csv/json/jsonl) via CLI or API
2. Reuse feature constructors
3. Load ensemble of XGBoost models + thresholds
4. Compute raw similarity scores, average across models
5. Apply threshold & matching strategy
6. Output similarity_matrix_value.csv, similarity_matrix_label.csv, and predicted pairs
a. Main Components
• User Entrypoints:
• Python API (schema_matching())
• CLI scripts (cal_column_similarity.py, train.py)
• Data Sources:
• Training Data folders (Table1.csv, Table2.csv, mapping.txt)
• Test Data folders (Table1.csv/.json, Table2.csv/.json)
• Feature Engineering Module:
• Column–level features (relation_features.py)
• Instance–level features (self_features.py)
• Model Training Module:
• XGBoost training engine (train.py)
• Model Storage:
• Versioned model directories containing .model and .threshold
• Inference Module:
• Scoring engine (cal_column_similarity.py)
• Strategy logic (one-to-one, one-to-many, many-to-many) in utils.py
• Output:
• CSVs for similarity values and labels
• Of matched column-pairs
b. Relationships & Data Flow
• User CLI/API → Data Loader → Feature Constructors → (Training: → XGBoost Trainer → Model Store)
• Inference: Feature Constructors + Model Loader → Ensemble Scorer → Threshold & Strategy → Outputs
c. Architectural Patterns & Principles
• Pipeline pattern: separate stages for data ingestion, feature extraction, model training/inference.
• Ensemble pattern: multiple XGBoost models with averaging.
• Separation of concerns: clear boundaries between feature logic, model logic, I/O, and strategy.
• Extensibility: ability to plug in new features or models by adding scripts and updating model folder.
d. Key Technologies & Libraries
• pandas, numpy for data manipulation
• nltk & python-dateutil for text/date features
• sentence-transformers for multilingual text embeddings (TSM cosine)
• xgboost for classification models
• strsimpy for string similarity metrics