How mildsauce9/stock-data-retrieval works
Project Type and Purpose
• This is a lightweight full-stack Retrieval-Augmented Generation (RAG) web application.
• Purpose: collect and preprocess financial news and stock data, embed and index it in Pinecone, then expose a web interface for users to enter queries that are answered by retrieving context from Pinecone and forwarding it to an LLM.
Key Architectural Components
a. Data Ingestion & Preprocessing
• JSON data files (e.g. Reuters_news_data.json, CNBC_news_data.json)
• Jupyter notebooks (load_news.ipynb, view_data.ipynb) for cleaning & exploratory analysis
• model.py orchestrates embedding of cleaned data using OpenAI’s text-embedding-3-small model
• requirements.txt lists dependencies (e.g. openai, pinecone-client)
b. Vector Storage
• Pinecone vector database to store upserted embeddings and metadata
c. Backend API
• app.py (Flask or FastAPI) exposes endpoints:
• GET “/” serves templates/index.html
• POST “/query” accepts user prompt
• Calls embedding API on query, retrieves top-k vectors from Pinecone, constructs final prompt, calls LLM, returns response
d. Frontend UI
• templates/index.html (HTML + minimal JS) – chat interface, displays user questions & bot answers
e. External Services
• OpenAI embedding and completion APIs
• Pinecone vector index
Relationships & Data Flow
1. Offline pipeline: raw JSON → notebooks → cleaned JSON → embedding (model.py) → upsert embeddings into Pinecone
2. Online user flow:
• User enters query in browser → HTTP POST → app.py
• Backend: embed query → Pinecone query → get nearest news context → build LLM prompt → call OpenAI → receive answer → HTTP response → UI displays answer
Architectural Patterns & Design Principles
• Separation of concerns: offline data pipeline vs. real-time serving
• Modularization: model.py for embedding logic, app.py for web layer
• Stateless backend: each request is independent, relies on external state (Pinecone)
• Scalability point: Pinecone handles vector search, OpenAI handles LLM compute
1. Start by laying out the “Offline Pipeline” on the left:
• JSON files → Data Cleaning Notebook → Embedding Module → Pinecone
• Browser UI → Web Server → Embedding Module → Pinecone → Prompt Builder → OpenAI → Web Server → Browser UI