How hamedbabaei/llms4om works
Project type and purpose
• This is a Python-based machine-learning framework (no browser UI) for Ontology Matching (OM) that orchestrates data ingestion, encoding, retrieval/LLM-based matching, post-processing, and evaluation across multiple datasets.
• Key purpose: provide a modular, pipeline-driven toolkit (LLMs4OM) to compare source and target ontology concepts via retrieval and/or RAG approaches using lightweight encoders, vector retrievers, and LLMs.
High-level components
A. Data Layer
• Ontology files & datasets (datasets/*)
• Pre-computed embeddings & assets (assets/)
• Result & visualization storage (experiments/, notebooks/)
B. Configuration & Environment
• BaseConfig (.env, .env-example, requirements.txt)
• Dockerfile / docker_builder.sh
C. Core Library Modules (ontomap package)
1. Ontology parsers (ontomap/ontology)
2. Encoders (ontomap/encoder: lightweight, RAG, fewshot, etc.)
3. Matchers (ontomap/ontology_matchers)
– Retrieval models (TFIDF, BERT, Specter, Ada…)
– Hybrid RAG models (OpenAI, LLaMA, Mistral + retriever)
– Other algorithms (ICV, few-shot, lightweight)
4. Pipeline orchestration (ontomap/pipeline/OMPipeline)
5. Post-processing (ontomap/postprocess)
6. Evaluation & metrics (ontomap/evaluation)
7. Utilities & I/O (ontomap/utils, ontomap/tools)
D. Experimentation & Automation
• Entrypoint scripts (scripts/*.sh, .py)
• Jupyter notebooks for analysis (notebooks/)
• Experiment artifacts (experiments/outputs, results, tables, visualizations)
Core data flow and interactions
1. Configuration load: BaseConfig reads CLI args/.env → initializes hyper-parameters (approach, encoder, thresholds, device, paths).
2. Data ingestion: Ontology parser loads source/target ontologies + reference alignments.
3. Encoding: Chosen encoder transforms each source concept (and optionally context: parents/children) into query vectors or prompt-ready text.
4. Matching (branch by approach):
• Retrieval only: encoder → vector DB (in-memory FAISS or similar) → retriever model scores top-k target concepts.
• RAG/hybrid: same retrieval step + assemble LLM prompt pairs → call LLM API (OpenAI, LLaMA locally) → LLM returns match probability/classification.
5. Post-processing: Apply thresholds (ir_score, llm_confidence), filter/aggregate results.
6. Evaluation: Compare predictions vs. ground truth → compute precision/recall/F1 via evaluator.
7. Results export: Write CSV/LaTeX tables, pivot visualizations, log to experiments/.
Architectural patterns & design principles
• Pipeline pattern: each stage (ingest → encode → match → postprocess → eval) is a pluggable module.
• Strategy/Factory pattern: different encoder and matcher classes implement a common interface (generate(input_data)).
• Separation of concerns: parsing, encoding, matching, post-processing, evaluation live in separate subpackages.
• Config-driven: switching approaches, encoders, devices via BaseConfig.
• Extensibility: new matcher or encoder simply registers under ontology_matchers or encoder.