How montagebai/kgfm works
Project Type and Purpose
• This is a data‐processing and knowledge‐graph construction pipeline (KG‐FM) built in Python.
• It consumes scientific literature (text and JSON metadata), extracts structured information about framework materials (MOFs, COFs, HOFs) using LLMs, and populates a Neo4j graph database.
• It also provides knowledge retrieval and question‐answering services via integration with the Qwen2 LLM.
Key File‐Structure Observations
• Top‐level resource directories:
• “Abstract Text” (zip archives of raw abstracts)
• “Json File” (zip archives of JSON metadata)
• Data artifacts: “Question and Answer.csv”
• Processing scripts (all Python):
• split_txt.py – splits raw text into manageable chunks
• alitest.py – likely an automated extraction/testing harness
• kg+llm_basic.py – QA service glue between KG and LLM
• titletoneo4j.py & toneo4j.py – ingestion of parsed data into Neo4j
• demonstrating video (for users)
• README outlines project goals, dependencies (Python, Neo4j, Qwen2), and use cases (graph construction, retrieval, mining, QA).
Main Components
a. Data Ingestion Layer
• Raw Data Sources: Abstract zip archives, JSON zip archives, CSV Q/A files
• Scripts: split_txt.py
b. Information Extraction Layer
• LLM‐based extraction: alitest.py (or equivalent modules calling Qwen2)
• Parsing modules: custom parsers converting text/JSON to intermediate objects
c. Graph Construction Layer
• Neo4j Database (hosted locally or on server)
• Ingestion scripts: titletoneo4j.py, toneo4j.py
d. Knowledge Retrieval & QA Service
• kg+llm_basic.py that (1) accepts user queries, (2) issues Cypher queries to Neo4j, (3) sends context + questions to Qwen2, (4) returns answers with citations
e. User Interface
• CLI or REST endpoint (if any) wrapping kg+llm_basic.py
• Demonstration video showing usage
f. External Dependencies
• Qwen2 LLM (external or local model)
• Neo4j database instance
Relationships and Data Flows
Raw Archives → split_txt.py → chunked text
chunked text → alitest.py → extracted entities/relations
JSON archives → custom parsers → intermediate data
intermediate data → titletoneo4j.py/toneo4j.py → Neo4j
Neo4j ←→ kg+llm_basic.py → Qwen2 model
User CLI → kg+llm_basic.py → returns answers
Architectural Patterns & Principles
• ETL pipeline pattern (Extract → Transform → Load)
• Separation of concerns: ingestion, extraction, storage, serving
• Modular scripts for each stage to facilitate testing and maintenance
• Use of a graph‐database microservice (Neo4j) distinct from processing code
• Integration with an external LLM service for NLP tasks