How lithuanianmathemator/neo4jdemo works
Project Type and Purpose
• This is a Python-based data processing and ML pipeline (no UI) whose goal is to:
• Ingest textbook PDFs
• Parse/extract text
• Construct a Neo4j knowledge graph (KG) with nodes, edges, and embeddings
• Run a baseline RAG approach (using Postgres as a simple vector store)
• Compare Q&A performance between the KG-based and RAG-based methods
File Structure
• Top-level scripts:
• main.py – orchestrates end-to-end pipeline
• infer.py – runs Q&A/inference
• “parsing/” – PDF→text pipeline (extract_page.py, parser.py, convert_csv.py)
• “KG/” – KG construction and embedding logic (create_nodes_layered.py, create_keywords.py, embedding.py, neo.py)
• “postgresDB/” – baseline RAG vector loading & query (postgre_load.py, query.py)
• data/ – source PDFs, intermediate outputs
• .env.example, requirements.txt – external credentials & dependencies
Extract Architectural Insights from README
• Two parallel query flows:
A. Knowledge-Graph Q&A via Cypher + embeddings + LLM
B. Naïve RAG via vector retrieval from Postgres + LLM
• External services:
• Neo4j database
• Postgres database
• OpenAI (or another LLM) for embeddings & completion
Main Components
a. Data Ingestion & Parsing Layer
• Component: “PDF Source” (data/*.pdf)
• “Parsing Service” (parsing/*.py) outputs markdown/CSV
b. KG Construction Layer
• “Node/Edge Creator” (KG/create_*.py)
• “Embedding Service” (KG/embedding.py calling OpenAI)
• “Neo4j Client” (KG/neo.py) injects to Neo4j
c. Baseline RAG Layer
• “VectorStore Loader” (postgresDB/postgre_load.py)
• “VectorStore Query” (postgresDB/query.py)
d. Inference & Q&A Layer
• “Infer Engine” (infer.py)
• Subflow 2: Query Postgres → retrieve top-K vectors → pass to LLM → answer
e. External Dependencies
• Neo4j database instance
• Postgres database instance
• OpenAI (or chosen LLM/embedding API)
Relationships & Data Flows
• PDF Source → Parsing Service → intermediate markdown
• markdown → KG Construction → Neo4j
• markdown → Embedding Service → Postgres (vector store)
• User Query → Infer Engine
• → Neo4j branch (Cypher + embeddings) → LLM → KG Answer
• → Postgres branch (vector retrieval) → LLM → RAG Answer
• Compare answers for performance evaluation
Architectural Patterns & Principles
• Modular pipelines (ingest, parse, build, query)
• Parallel comparison flows (KG vs. RAG)
• Separation of concerns: graph DB vs. vector DB
• Use of micro-libraries for each function (parser, KG, Postgres, infer)