# montagebai/kgfm architecture

> Architecture diagram of the GitHub repository montagebai/kgfm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/montagebai/kgfm
- Repository: https://github.com/montagebai/kgfm

## Overview

##### Project Type and Purpose
- This is a data‐processing and knowledge‐graph construction pipeline (KG‐FM) built in Python.
- It consumes scientific literature (text and JSON metadata), extracts structured information about framework materials (MOFs, COFs, HOFs) using LLMs, and populates a Neo4j graph database.
- It also provides knowledge retrieval and question‐answering services via integration with the Qwen2 LLM.

##### Key File‐Structure Observations
- Top‐level resource directories:
- “Abstract Text” (zip archives of raw abstracts)
- “Json File” (zip archives of JSON metadata)
- Data artifacts: “Question and Answer.csv”
- Processing scripts (all Python):
- split_txt.py – splits raw text into manageable chunks
- alitest.py – likely an automated extraction/testing harness
- kg+llm_basic.py – QA service glue between KG and LLM
- titletoneo4j.py & toneo4j.py – ingestion of parsed data into Neo4j
- demonstrating video (for users)
- README outlines project goals, dependencies (Python, Neo4j, Qwen2), and use cases (graph construction, retrieval, mining, QA).

##### Main Components
a. Data Ingestion Layer
- Raw Data Sources: Abstract zip archives, JSON zip archives, CSV Q/A files
- Scripts: split_txt.py
b. Information Extraction Layer
- LLM‐based extraction: alitest.py (or equivalent modules calling Qwen2)
- Parsing modules: custom parsers converting text/JSON to intermediate objects
c. Graph Construction Layer
- Neo4j Database (hosted locally or on server)
- Ingestion scripts: titletoneo4j.py, toneo4j.py
d. Knowledge Retrieval & QA Service
- kg+llm_basic.py that (1) accepts user queries, (2) issues Cypher queries to Neo4j, (3) sends context + questions to Qwen2, (4) returns answers with citations
e. User Interface
- CLI or REST endpoint (if any) wrapping kg+llm_basic.py
- Demonstration video showing usage
f. External Dependencies
- Qwen2 LLM (external or local model)
- Neo4j database instance

##### Relationships and Data Flows
Raw Archives → split_txt.py → chunked text
chunked text → alitest.py → extracted entities/relations
JSON archives → custom parsers → intermediate data
intermediate data → titletoneo4j.py/toneo4j.py → Neo4j
Neo4j ←→ kg+llm_basic.py → Qwen2 model
User CLI → kg+llm_basic.py → returns answers

##### Architectural Patterns & Principles
- ETL pipeline pattern (Extract → Transform → Load)
- Separation of concerns: ingestion, extraction, storage, serving
- Modular scripts for each stage to facilitate testing and maintenance
- Use of a graph‐database microservice (Neo4j) distinct from processing code
- Integration with an external LLM service for NLP tasks

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        COFAbstract["COF Abstract ZIP"]:::data
        HOFAbstract["HOF Abstract ZIP"]:::data
        MOFAbstract["MOF Abstract ZIP"]:::data
        COFJson["COF JSON ZIP"]:::data
        HOFJson["HOF JSON ZIP"]:::data
        MOFJson["MOF JSON ZIP"]:::data
        QAData["Question and Answer.csv"]:::data
        Splitter["split_txt.py\n(Text Chunker)"]:::script
    end

    %% Information Extraction Layer
    subgraph "Information Extraction Layer"
        direction TB
        Extractor["alitest.py\n(LLM Extraction Harness)"]:::script
    end

    %% Graph Construction Layer
    subgraph "Graph Construction Layer"
        direction TB
        TitleIngest["titletoneo4j.py\n(Title-based Ingestion)"]:::script
        GeneralIngest["toneo4j.py\n(General Ingestion)"]:::script
        Neo4jDB["Neo4j Database\n(2.5M+ nodes, 4M+ rels)"]:::db
    end

    %% Knowledge Retrieval & QA Service
    subgraph "Knowledge Retrieval & QA Service"
        direction TB
        KGLLM["kg+llm_basic.py\n(KG⇄LLM Glue)"]:::script
        Qwen2["Qwen2 LLM"]:::external
    end

    %% User Interface Layer
    subgraph "User Interface"
        direction TB
        CLI["CLI / REST Interface"]:::ui
        DemoVid["demonstration video.mp4"]:::ui
        README["README.md"]:::ui
    end

    %% Data Flows
    COFAbstract -->|raw abstracts| Splitter
    HOFAbstract -->|raw abstracts| Splitter
    MOFAbstract -->|raw abstracts| Splitter
    COFJson -->|raw JSON| Extractor
    HOFJson -->|raw JSON| Extractor
    MOFJson -->|raw JSON| Extractor
    QAData -->|reference data| Extractor
    Splitter -->|text chunks| Extractor
    Extractor -->|parsed entities| TitleIngest
    Extractor -->|parsed entities| GeneralIngest
    TitleIngest -->|Cypher write| Neo4jDB
    GeneralIngest -->|Cypher write| Neo4jDB
    CLI -->|user query| KGLLM
    KGLLM -->|Cypher query| Neo4jDB
    Neo4jDB -->|query result| KGLLM
    KGLLM -->|LLM prompt/response| Qwen2
    DemoVid --> CLI
    README --> CLI

    %% Click Events
    click COFAbstract "https://github.com/montagebai/kgfm/blob/main/Abstract Text/COFAbstracttext.zip"
    click HOFAbstract "https://github.com/montagebai/kgfm/blob/main/Abstract Text/HOFAbstracttext.zip"
    click MOFAbstract "https://github.com/montagebai/kgfm/blob/main/Abstract Text/MOFAbstracttext.zip"
    click COFJson "https://github.com/montagebai/kgfm/blob/main/Json File/COFJsonfile.zip"
    click HOFJson "https://github.com/montagebai/kgfm/blob/main/Json File/HOFJsonfile.zip"
    click MOFJson "https://github.com/montagebai/kgfm/blob/main/Json File/MOFJsonfile.zip"
    click QAData "https://github.com/montagebai/kgfm/blob/main/Question and Answer.csv"
    click Splitter "https://github.com/montagebai/kgfm/blob/main/split_txt.py"
    click Extractor "https://github.com/montagebai/kgfm/blob/main/alitest.py"
    click TitleIngest "https://github.com/montagebai/kgfm/blob/main/titletoneo4j.py"
    click GeneralIngest "https://github.com/montagebai/kgfm/blob/main/toneo4j.py"
    click KGLLM "https://github.com/montagebai/kgfm/blob/main/kg+llm_basic.py"
    click DemoVid "https://github.com/montagebai/kgfm/blob/main/demonstration video.mp4"
    click README "https://github.com/montagebai/kgfm/blob/main/README.md"

    %% Styles
    classDef data fill:#FFA500,stroke:#333,stroke-width:1px
    classDef script fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef db fill:#98FB98,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef ui fill:#D8BFD8,stroke:#333,stroke-width:1px
```
