# lithuanianmathemator/neo4jdemo architecture

> Architecture diagram of the GitHub repository lithuanianmathemator/neo4jdemo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lithuanianmathemator/neo4jdemo
- Repository: https://github.com/lithuanianmathemator/neo4jdemo

## Overview

##### Project Type and Purpose
- This is a Python-based data processing and ML pipeline (no UI) whose goal is to:
- Ingest textbook PDFs
- Parse/extract text
- Construct a Neo4j knowledge graph (KG) with nodes, edges, and embeddings
- Run a baseline RAG approach (using Postgres as a simple vector store)
- Compare Q&A performance between the KG-based and RAG-based methods

##### File Structure
- Top-level scripts:
- main.py – orchestrates end-to-end pipeline
- infer.py – runs Q&A/inference
- “parsing/” – PDF→text pipeline (extract_page.py, parser.py, convert_csv.py)
- “KG/” – KG construction and embedding logic (create_nodes_layered.py, create_keywords.py, embedding.py, neo.py)
- “postgresDB/” – baseline RAG vector loading & query (postgre_load.py, query.py)
- data/ – source PDFs, intermediate outputs
- .env.example, requirements.txt – external credentials & dependencies

##### Extract Architectural Insights from README
- Two parallel query flows:
A. Knowledge-Graph Q&A via Cypher + embeddings + LLM
B. Naïve RAG via vector retrieval from Postgres + LLM
- External services:
- Neo4j database
- Postgres database
- OpenAI (or another LLM) for embeddings & completion

##### Main Components
a. Data Ingestion & Parsing Layer
- Component: “PDF Source” (data/*.pdf)
- “Parsing Service” (parsing/*.py) outputs markdown/CSV
b. KG Construction Layer
- “Node/Edge Creator” (KG/create_*.py)
- “Embedding Service” (KG/embedding.py calling OpenAI)
- “Neo4j Client” (KG/neo.py) injects to Neo4j
c. Baseline RAG Layer
- “VectorStore Loader” (postgresDB/postgre_load.py)
- “VectorStore Query” (postgresDB/query.py)
d. Inference & Q&A Layer
- “Infer Engine” (infer.py)
- Subflow 2: Query Postgres → retrieve top-K vectors → pass to LLM → answer
e. External Dependencies
- Neo4j database instance
- Postgres database instance
- OpenAI (or chosen LLM/embedding API)

##### Relationships & Data Flows
- PDF Source → Parsing Service → intermediate markdown
- markdown → KG Construction → Neo4j
- markdown → Embedding Service → Postgres (vector store)
- User Query → Infer Engine
- → Neo4j branch (Cypher + embeddings) → LLM → KG Answer
- → Postgres branch (vector retrieval) → LLM → RAG Answer
- Compare answers for performance evaluation

##### Architectural Patterns & Principles
- Modular pipelines (ingest, parse, build, query)
- Parallel comparison flows (KG vs. RAG)
- Separation of concerns: graph DB vs. vector DB
- Use of micro-libraries for each function (parser, KG, Postgres, infer)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion & Parsing Layer
    subgraph "Data Ingestion & Parsing"
        direction TB
        PDF["PDF Source"]:::ingestion
        Parsing["Parsing Service"]:::ingestion
        Intermediate["Intermediate Data"]:::ingestion
    end

    %% KG Construction Layer
    subgraph "KG Construction" 
        direction TB
        NodeCreator["Create Nodes & Edges"]:::processing
        KeywordCreator["Create Keywords"]:::processing
        EmbeddingService["Embedding Service"]:::processing
        Neo4jClient["Neo4j Client"]:::processing
    end

    %% Baseline RAG Layer
    subgraph "Baseline RAG"
        direction TB
        VectorLoader["VectorStore Loader"]:::processing
        VectorQuery["VectorStore Query"]:::processing
    end

    %% Orchestration & Inference
    subgraph "Orchestration & Inference"
        direction TB
        Main["Pipeline Orchestrator"]:::processing
        Infer["Infer Engine"]:::inference
    end

    %% External Services & DBs
    subgraph "External Services & Databases"
        direction TB
        Neo4jDB["Neo4j Database"]:::external
        PostgresDB["Postgres Database"]:::external
        LLM["OpenAI LLM"]:::external
        User["User Query"]:::external
        KGAnswer["KG Answer"]:::external
        RAGAnswer["RAG Answer"]:::external
    end

    %% Flows
    Main --> PDF
    Main --> Parsing
    Main --> NodeCreator
    Main --> VectorLoader

    PDF --> Parsing --> Intermediate
    Intermediate --> NodeCreator --> Neo4jClient --> Neo4jDB
    Intermediate --> EmbeddingService --> VectorLoader --> PostgresDB

    User --> Infer

    Infer -->|"Cypher Query"| Neo4jDB
    Neo4jDB -->|"subgraph+embeddings"| LLM
    LLM -->|"KG Answer"| KGAnswer

    Infer -->|"Vector Query"| PostgresDB
    PostgresDB -->|"vectors"| LLM
    LLM -->|"RAG Answer"| RAGAnswer

    %% Click Events
    click PDF "https://github.com/lithuanianmathemator/neo4jdemo/tree/main/data/"
    click Parsing "https://github.com/lithuanianmathemator/neo4jdemo/tree/main/parsing/"
    click NodeCreator "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/KG/create_nodes_layered.py"
    click KeywordCreator "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/KG/create_keywords.py"
    click EmbeddingService "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/KG/embedding.py"
    click Neo4jClient "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/KG/neo.py"
    click VectorLoader "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/postgresDB/postgre_load.py"
    click VectorQuery "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/postgresDB/query.py"
    click Main "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/main.py"
    click Infer "https://github.com/lithuanianmathemator/neo4jdemo/blob/main/infer.py"

    %% Styles
    classDef ingestion fill:#D6E9FF,stroke:#005599,color:#000;
    classDef processing fill:#FCF8E3,stroke:#8A6D3B,color:#000;
    classDef inference fill:#FDEBD0,stroke:#E67E22,color:#000;
    classDef external fill:#F5F5F5,stroke:#999,color:#000;
```
