# ai4chems/mof_chemunity architecture

> Architecture diagram of the GitHub repository ai4chems/mof_chemunity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ai4chems/mof_chemunity
- Repository: https://github.com/ai4chems/mof_chemunity

## Overview

#### Purpose and entry points
MOF-ChemUnity builds and queries a Neo4j knowledge graph linking MOF computational data with LLM-extracted literature facts through CSD codes and DOI. Primary user entry point is `QueryGenerationAgent.run_full_query()` in `src/MOF_ChemUnity/Agents/QueryAgent.py`. Dataset extension is notebook-led: `src/ChemUnity_Matching.ipynb` then `src/ChemUnity_Extraction.ipynb`; graph loading is `src/Knowledge Graph Creation/neo4j_import.py`.

#### Agent layer
`BaseAgent` provides the shared agent foundation, specialized by extraction, entity matching, and natural-language graph querying:
- `src/MOF_ChemUnity/Agents/BaseAgent.py`
- `src/MOF_ChemUnity/Agents/ExtractionAgent.py`
- `src/MOF_ChemUnity/Agents/MatchingAgent.py`
- `src/MOF_ChemUnity/Agents/QueryAgent.py`

This separates LLM-driven interpretation from data preparation and graph persistence.

#### Literature ingestion and normalization
Documents are converted into usable text/Markdown before extraction. Extraction agents produce structured information such as MOF names, properties, applications, synthesis, and water stability; matching connects literature mentions to CSD identifiers. Prompt modules define task-specific LLM instructions:
- `src/MOF_ChemUnity/utils/DocProcessor.py`
- `src/MOF_ChemUnity/utils/XML_to_MD.py`
- `src/MOF_ChemUnity/Prompts/Extraction_Prompts.py`
- `src/MOF_ChemUnity/Prompts/Matching_Prompts.py`

Example PDF/XML inputs and derived Markdown demonstrate the ingestion boundary under `src/Examples/PDF/` and `src/Examples/XML/`.

#### Data quality, identity, and enrichment
Utilities prepare, filter, model, and reconcile extracted records before graph import. The CSD code is the central identity boundary joining paper-derived facts to computational sources; MOF/CSD reference data supports this linkage.
- `src/MOF_ChemUnity/utils/DataModels.py`
- `src/MOF_ChemUnity/utils/DataPrep.py`
- `src/MOF_ChemUnity/utils/FilterTools.py`
- `src/MOF_ChemUnity/utils/csd_dict.py`
- `data/MOF_names_and_CSD_codes.csv`

#### Knowledge-graph persistence
Neo4j is the material runtime and system of record. Import expects relational CSV exports for matching, experimental properties, applications, synthesis, water stability, descriptors, and computational properties, then creates the graph and relationships:
- `src/Knowledge Graph Creation/neo4j_import.py`
- `src/Examples/KG_Data/matching.csv`
- `src/Examples/KG_Data/computational_properties.csv`
- `src/Examples/KG_Data/filtered_experimental_properties.csv`

Connection credentials are supplied through `NEO4J_URI`, `NEO4J_USER`, and `NEO4J_PASSWORD`, keeping infrastructure access outside package code.

#### Query and retrieval flow
A user question enters `QueryGenerationAgent`, which translates it into Neo4j-oriented retrieval and returns a Pandas DataFrame. Query prompts are isolated from the agent implementation:
- `src/MOF_ChemUnity/Agents/QueryAgent.py`
- `src/MOF_ChemUnity/Prompts/Query_Generation_Prompts.py`

Semantic/similarity retrieval artifacts use FAISS indexes, with neighboring-record logic in:
- `src/MOF_ChemUnity/utils/NeighbourFinder.py`
- `src/Examples/vs/10.1016_j.molstruc.2017.11.128/index.faiss`

#### External services and expansion
OpenAI/GPT-4o is the intended LLM service for extraction and query generation; users provide their own API key. Neo4j hosts the graph. Cross-document expansion utilities support finding related literature and appending new extractions:
- `src/MOF_ChemUnity/utils/cross_doc_utils.py`
- `src/Cross_Document.ipynb`

Main entry points: [`src/MOF_ChemUnity/Agents/ExtractionAgent.py`](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/ExtractionAgent.py) (Extraction agent), [`src/MOF_ChemUnity/Agents/MatchingAgent.py`](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/MatchingAgent.py) (Matching agent), [`src/MOF_ChemUnity/Agents/QueryAgent.py`](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/QueryAgent.py) (Query generation agent), [`src/MOF_ChemUnity/Agents/BaseAgent.py`](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/BaseAgent.py) (Base agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Literature ingestion"]
  node_source_docs["PDF/XML literature<br/>source documents"]
  node_doc_processor["Document processing<br/>text normalization<br/>[DocProcessor.py]"]
  node_xml_to_md["XML to Markdown<br/>converter<br/>[XML_to_MD.py]"]
end

subgraph group_agents["Agent layer"]
  node_extraction_agent["Extraction agent<br/>LLM extraction<br/>[ExtractionAgent.py]"]
  node_matching_agent["Matching agent<br/>entity matching<br/>[MatchingAgent.py]"]
  node_base_agent["Base agent<br/>agent foundation<br/>[BaseAgent.py]"]
  node_extraction_prompts["Extraction prompts<br/>LLM prompts"]
  node_matching_prompts["Matching prompts<br/>LLM prompts"]
end

subgraph group_data["Data preparation"]
  node_record_prep["Record preparation<br/>data quality<br/>[DataPrep.py]"]
  node_csd_reference["MOF/CSD reference<br/>identity reference"]
  node_cross_document["Cross-document expansion<br/>literature expansion<br/>[cross_doc_utils.py]"]
end

subgraph group_graph["Knowledge graph"]
  node_csv_exports["Relational CSV exports<br/>graph import data<br/>[matching.csv]"]
  node_neo4j_import["Neo4j importer<br/>graph loader<br/>[neo4j_import.py]"]
  node_neo4j[("Neo4j knowledge graph<br/>system of record")]
end

subgraph group_query["Query and similarity"]
  node_query_agent["Query generation agent<br/>natural-language query<br/>[QueryAgent.py]"]
  node_query_prompts["Query prompts<br/>LLM prompts"]
  node_faiss[("FAISS similarity index<br/>semantic retrieval<br/>[index.faiss]")]
end

node_openai{{"OpenAI GPT-4o<br/>external LLM"}}

node_source_docs -->|"normalize"| node_doc_processor
node_source_docs -->|"convert XML"| node_xml_to_md
node_doc_processor -->|"text/Markdown"| node_extraction_agent
node_xml_to_md -->|"Markdown"| node_extraction_agent
node_base_agent -->|"specializes"| node_extraction_agent
node_base_agent -->|"specializes"| node_matching_agent
node_base_agent -->|"specializes"| node_query_agent
node_extraction_prompts -->|"instructions"| node_extraction_agent
node_matching_prompts -->|"instructions"| node_matching_agent
node_extraction_agent -->|"LLM calls"| node_openai
node_matching_agent -->|"LLM calls"| node_openai
node_extraction_agent -->|"structured facts"| node_record_prep
node_record_prep -->|"normalized records"| node_matching_agent
node_csd_reference -->|"CSD lookup"| node_matching_agent
node_matching_agent -->|"matched records"| node_csv_exports
node_record_prep -->|"filtered facts"| node_csv_exports
node_cross_document -.->|"related documents"| node_extraction_agent
node_csv_exports -->|"load input"| node_neo4j_import
node_neo4j_import -->|"create graph"| node_neo4j
node_query_prompts -->|"instructions"| node_query_agent
node_query_agent -->|"generate retrieval"| node_openai
node_query_agent -->|"Neo4j retrieval"| node_neo4j
node_faiss -.->|"similarity context"| node_query_agent

click node_doc_processor "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/DocProcessor.py"
click node_xml_to_md "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/XML_to_MD.py"
click node_extraction_agent "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/ExtractionAgent.py"
click node_matching_agent "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/MatchingAgent.py"
click node_base_agent "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/BaseAgent.py"
click node_extraction_prompts "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Prompts/Extraction_Prompts.py"
click node_matching_prompts "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Prompts/Matching_Prompts.py"
click node_record_prep "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/DataPrep.py"
click node_csd_reference "https://github.com/ai4chems/mof_chemunity/blob/main/data/MOF_names_and_CSD_codes.csv"
click node_cross_document "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/cross_doc_utils.py"
click node_csv_exports "https://github.com/ai4chems/mof_chemunity/blob/main/src/Examples/KG_Data/matching.csv"
click node_neo4j_import "https://github.com/ai4chems/mof_chemunity/blob/main/src/Knowledge%20Graph%20Creation/neo4j_import.py"
click node_query_agent "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/QueryAgent.py"
click node_query_prompts "https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Prompts/Query_Generation_Prompts.py"
click node_faiss "https://github.com/ai4chems/mof_chemunity/blob/main/src/Examples/vs/10.1016_j.molstruc.2017.11.128/index.faiss"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_source_docs,node_doc_processor,node_xml_to_md toneBlue
class node_extraction_agent,node_matching_agent,node_base_agent,node_extraction_prompts,node_matching_prompts toneAmber
class node_record_prep,node_csd_reference,node_cross_document toneMint
class node_csv_exports,node_neo4j_import,node_neo4j toneRose
class node_query_agent,node_query_prompts,node_faiss toneIndigo
class node_openai toneNeutral
```

## Components

### Literature ingestion

- **PDF/XML literature**: source documents · Example inputs and derived Markdown establish the ingestion boundary.
- [Document processing](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/DocProcessor.py): text normalization · `src/MOF_ChemUnity/utils/DocProcessor.py`
- [XML to Markdown](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/XML_to_MD.py): converter · `src/MOF_ChemUnity/utils/XML_to_MD.py`

### Agent layer

- [Extraction agent](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/ExtractionAgent.py): LLM extraction · Extracts MOF facts including properties, applications, synthesis, and stability. · `src/MOF_ChemUnity/Agents/ExtractionAgent.py`
- [Matching agent](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/MatchingAgent.py): entity matching · Links literature mentions to CSD identifiers. · `src/MOF_ChemUnity/Agents/MatchingAgent.py`
- [Base agent](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/BaseAgent.py): agent foundation · Shared foundation for specialized agents. · `src/MOF_ChemUnity/Agents/BaseAgent.py`
- [Extraction prompts](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Prompts/Extraction_Prompts.py): LLM prompts · `src/MOF_ChemUnity/Prompts/Extraction_Prompts.py`
- [Matching prompts](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Prompts/Matching_Prompts.py): LLM prompts · `src/MOF_ChemUnity/Prompts/Matching_Prompts.py`

### Data preparation

- [Record preparation](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/DataPrep.py): data quality · Models, prepares, filters, and reconciles extracted records. · `src/MOF_ChemUnity/utils/DataPrep.py`
- [MOF/CSD reference](https://github.com/ai4chems/mof_chemunity/blob/main/data/MOF_names_and_CSD_codes.csv): identity reference · CSD code is the join boundary between literature and computational data. · `data/MOF_names_and_CSD_codes.csv`
- [Cross-document expansion](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/utils/cross_doc_utils.py): literature expansion · Finds related literature and appends new extractions. · `src/MOF_ChemUnity/utils/cross_doc_utils.py`

### Knowledge graph

- [Relational CSV exports](https://github.com/ai4chems/mof_chemunity/blob/main/src/Examples/KG_Data/matching.csv): graph import data · Includes matching, experimental, computational, and literature-derived facts. · `src/Examples/KG_Data/matching.csv`
- [Neo4j importer](https://github.com/ai4chems/mof_chemunity/blob/main/src/Knowledge%20Graph%20Creation/neo4j_import.py): graph loader · Creates graph nodes and relationships from CSV exports. · `src/Knowledge Graph Creation/neo4j_import.py`
- **Neo4j knowledge graph**: system of record · Persists linked MOF computational and literature knowledge.

### Query and similarity

- [Query generation agent](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Agents/QueryAgent.py): natural-language query · Primary user entry point: run_full_query() returns a Pandas DataFrame. · `src/MOF_ChemUnity/Agents/QueryAgent.py`
- [Query prompts](https://github.com/ai4chems/mof_chemunity/blob/main/src/MOF_ChemUnity/Prompts/Query_Generation_Prompts.py): LLM prompts · `src/MOF_ChemUnity/Prompts/Query_Generation_Prompts.py`
- [FAISS similarity index](https://github.com/ai4chems/mof_chemunity/blob/main/src/Examples/vs/10.1016_j.molstruc.2017.11.128/index.faiss): semantic retrieval · Supports neighboring-record similarity retrieval. · `src/Examples/vs/10.1016_j.molstruc.2017.11.128/index.faiss`

### Other components

- **OpenAI GPT-4o**: external LLM · User-supplied API key enables extraction and query generation.

## Connections

- PDF/XML literature → Document processing: normalize
- PDF/XML literature → XML to Markdown: convert XML
- Document processing → Extraction agent: text/Markdown
- XML to Markdown → Extraction agent: Markdown
- Base agent → Extraction agent: specializes
- Base agent → Matching agent: specializes
- Base agent → Query generation agent: specializes
- Extraction prompts → Extraction agent: instructions
- Matching prompts → Matching agent: instructions
- Extraction agent → OpenAI GPT-4o: LLM calls
- Matching agent → OpenAI GPT-4o: LLM calls
- Extraction agent → Record preparation: structured facts
- Record preparation → Matching agent: normalized records
- MOF/CSD reference → Matching agent: CSD lookup
- Matching agent → Relational CSV exports: matched records
- Record preparation → Relational CSV exports: filtered facts
- Cross-document expansion → Extraction agent: related documents
- Relational CSV exports → Neo4j importer: load input
- Neo4j importer → Neo4j knowledge graph: create graph
- Query prompts → Query generation agent: instructions
- Query generation agent → OpenAI GPT-4o: generate retrieval
- Query generation agent → Neo4j knowledge graph: Neo4j retrieval
- FAISS similarity index → Query generation agent: similarity context