How jaygupta8/openai works
Project type and purpose
• This is essentially a data-processing/QA pipeline implemented in a Jupyter notebook.
• Purpose: load a PDF on Vedic Astrology, break it into chunks, embed each chunk with OpenAI, index embeddings in Pinecone, then answer user questions via semantic search.
Main components
a. Notebook Orchestrator
• Represents the vedic_astrology.ipynb itself, which sequences all steps.
b. Data Source
• PDF Book (e.g., a file in storage).
c. Text Splitter
• RecursiveCharacterTextSplitter from LangChain.
d. Embedding Generator
• OpenAI embeddings API.
e. Vector Store
• Pinecone index (external vector database).
f. Question Answering Chain
• OpenAI completion/chat API wrapped by LangChain ConversationChain.
g. User Interface
• Jupyter Notebook cell prompts (or potential upstream application).
Relationships and data flow
• PDF Book → Notebook Orchestrator: load raw PDF bytes.
• Notebook Orchestrator → Text Splitter: chunk text into Document objects.
• Text Splitter → Embedding Generator: send each chunk to OpenAI embeddings API.
• Embedding Generator → Vector Store: upsert vectors into Pinecone index.
• User Question → Notebook Orchestrator → Pinecone similarity search → retrieves top-k chunks.
• Retrieved chunks + question → QA Chain → OpenAI chat/completion API → answer returned back to user.
Architectural patterns and principles
• Linear data pipeline: each step feeds the next in sequence.
• External service integration: embedding and QA sub-systems are decoupled via vector store.
• Modularity: each logical step (loading, splitting, embedding, indexing, querying) is a distinct component and can be replaced or scaled independently.
Relevant technologies & libraries
• LangChain (PDF loader, splitter, chains)
• OpenAI API (embeddings, chat/completion)
• Pinecone (vector database)
• Jupyter Notebook (orchestrator & UI)