How gptechday/openai-academy-kg-recipe works
Project Type and Purpose
• This is a full-stack backend application focused on healthcare data processing. It leverages Synthea CSV healthcare data to build a knowledge graph, enriches it with vector indexing for semantic search, integrates with a Neo4j database, and exposes a FastAPI REST interface. Additionally, it incorporates natural language processing (NLP) through OpenAI models to translate user queries into actionable Cypher commands.
Analyzing the File Structure
• Key files and directories include:
• main.py – The entry point for the FastAPI server.
• utils/ – Contains modules like kg_builder.py (for constructing the graph), kg_to_vector.py (for vector indexing), neo4j_graph_builder.py (for integrating with Neo4j), synthea_ontology.py (which defines healthcare ontology), and cypher_writer.py (for generating Neo4j queries).
• prompts/ – Holds prompt files used with OpenAI models to process natural language queries.
• synthea_sample_data_csv_latest/ – Contains raw Synthea CSV datasets which are used to populate the knowledge graph.
• data/ – Likely used to store intermediary output such as the constructed knowledge graph (data/knowledge_graph.json).
• rebuild_index.py – A utility used to regenerate the vector index when modifications occur.
• Dockerfile and cloudbuild.yaml – For containerization and deployment.
• These components suggest clear divisions between data ingestion, graph construction, vector embedding creation, Cypher query generation, and API interfacing.
Core Components
a. API Layer:
• FastAPI application (main.py) exposing endpoints (e.g., /query and /rebuild_index).
b. Data Ingestion and Preprocessing:
• Synthea CSV data files (in synthea_sample_data_csv_latest).
• Data is read, preprocessed, and prepared for graph creation.
c. Knowledge Graph Construction:
• The utilities in utils/ (kg_builder.py, synthea_ontology.py) transform raw healthcare data into a structured graph.
d. Vector Indexing:
• The module kg_to_vector.py converts the knowledge graph into vector embeddings; rebuild_index.py triggers an update of this index.
e. Neo4j Integration:
• neo4j_graph_builder.py manages the interactions with the Neo4j database.
• cypher_writer.py handles translating natural language queries into Cypher queries.
f. NLP and Query Translation:
• The prompts in prompts/ work with OpenAI models to understand and convert natural language queries.
g. External Systems and Deployment:
• Neo4j Database: As the underlying graph database.
• OpenAI models: For natural language query processing.
• Docker container: For deployment and consistent runtime (Dockerfile).
• Cloud Build: Deployment automation (cloudbuild.yaml).
Relationships and Interactions
• API calls (especially /query) flow into the business logic where:
• The query is sent to the NLP processing module (using prompts/OpenAI models).
• The processed query is then converted to a Cypher command (cypher_writer.py) and executed against the Neo4j database (via neo4j_graph_builder.py).
• In parallel, vector search using the output of kg_to_vector.py may be performed to refine or support the query results.
• Data ingestion processes (reading Synthea CSV files) feed into the knowledge graph construction process.
• The vector indexing component interacts with the constructed graph, storing its summary in an index used during query resolution.
• Docker and cloud deployment components oversee containerization and continuous deployment flows.
Architectural Patterns and Technologies
• Component Separation: Dividing responsibilities across modules (API layer, data processing, graph building, NLP, and database integrations).
• Microservices-like Flow (even if within one service): Where data flows sequentially and components interact via well-defined interfaces.
• Use of REST APIs (via FastAPI) and containerization (via Docker) for deployment consistency.
• Leveraging external APIs (OpenAI’s language models) and databases (Neo4j).