# mildsauce9/stock-data-retrieval architecture

> Architecture diagram of the GitHub repository mildsauce9/stock-data-retrieval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mildsauce9/stock-data-retrieval
- Repository: https://github.com/mildsauce9/stock-data-retrieval

## Overview

##### Project Type and Purpose
- This is a lightweight full-stack Retrieval-Augmented Generation (RAG) web application.
- Purpose: collect and preprocess financial news and stock data, embed and index it in Pinecone, then expose a web interface for users to enter queries that are answered by retrieving context from Pinecone and forwarding it to an LLM.

##### Key Architectural Components
a. Data Ingestion & Preprocessing
- JSON data files (e.g. Reuters_news_data.json, CNBC_news_data.json)
- Jupyter notebooks (load_news.ipynb, view_data.ipynb) for cleaning & exploratory analysis
- model.py orchestrates embedding of cleaned data using OpenAI’s text-embedding-3-small model
- requirements.txt lists dependencies (e.g. openai, pinecone-client)
b. Vector Storage
- Pinecone vector database to store upserted embeddings and metadata
c. Backend API
- app.py (Flask or FastAPI) exposes endpoints:
- GET “/” serves templates/index.html
- POST “/query” accepts user prompt
- Calls embedding API on query, retrieves top-k vectors from Pinecone, constructs final prompt, calls LLM, returns response
d. Frontend UI
- templates/index.html (HTML + minimal JS) – chat interface, displays user questions & bot answers
e. External Services
- OpenAI embedding and completion APIs
- Pinecone vector index

##### Relationships & Data Flow
1. Offline pipeline: raw JSON → notebooks → cleaned JSON → embedding (model.py) → upsert embeddings into Pinecone
2. Online user flow:
- User enters query in browser → HTTP POST → app.py
- Backend: embed query → Pinecone query → get nearest news context → build LLM prompt → call OpenAI → receive answer → HTTP response → UI displays answer

##### Architectural Patterns & Design Principles
- Separation of concerns: offline data pipeline vs. real-time serving
- Modularization: model.py for embedding logic, app.py for web layer
- Stateless backend: each request is independent, relies on external state (Pinecone)
- Scalability point: Pinecone handles vector search, OpenAI handles LLM compute

1. Start by laying out the “Offline Pipeline” on the left:
- JSON files → Data Cleaning Notebook → Embedding Module → Pinecone
- Browser UI → Web Server → Embedding Module → Pinecone → Prompt Builder → OpenAI → Web Server → Browser UI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Offline Pipeline
    subgraph "Offline Pipeline"
        direction TB
        raw1[(Reuters_news_data.json)]:::datastore
        raw2[(CNBC_news_data.json)]:::datastore

        subgraph "Data Cleaning Notebooks"
            direction TB
            notebook1["load_news.ipynb"]:::internal
            notebook2["view_data.ipynb"]:::internal
        end

        cleaned1[(Cleaned_Reuters_news_data.json)]:::datastore
        cleaned2[(Cleaned_CNBC_news_data.json)]:::datastore

        embedding_off["Embedding Service\n(model.py & OpenAI Embedding API)"]:::internal
        pinecone[(Pinecone Index)]:::external

        raw1 -->|"raw text"| notebook1
        raw2 -->|"raw text"| notebook1
        notebook1 -->|"cleaned data"| cleaned1
        notebook1 -->|"cleaned data"| cleaned2
        cleaned1 -->|"text for embedding"| embedding_off
        cleaned2 -->|"text for embedding"| embedding_off
        embedding_off -->|"upsert vectors"| pinecone
    end

    %% Online Query Flow
    subgraph "Online Query Flow"
        direction TB
        browser["Browser UI\n(templates/index.html)"]:::ui
        web["Web Server\n(app.py)"]:::internal
        embed_on["Embedding Service\n(model.py)"]:::internal
        pinecone
        retrieved["Retrieved Context"]:::datastore
        prompt["Prompt Builder\n(app.py)"]:::internal
        llm[(OpenAI Completion API)]:::external
        response["LLM Response"]:::internal

        browser -->|"user query"| web
        web -->|"embed query"| embed_on
        embed_on -->|"query vectors"| pinecone
        pinecone -->|"nearest vectors"| retrieved
        retrieved -->|"context"| prompt
        prompt -->|"LLM prompt"| llm
        llm -->|"answer"| response
        response -->|"reply"| web
        web -->|"HTTP response"| browser
    end

    %% Auxiliary Data Store
    top_sum[(top_stock_summaries.json)]:::datastore

    %% Documentation
    req["requirements.txt"]:::internal
    readme["README.md"]:::internal

    %% Click Events
    click raw1 "https://github.com/mildsauce9/stock-data-retrieval/blob/main/Reuters_news_data.json"
    click raw2 "https://github.com/mildsauce9/stock-data-retrieval/blob/main/CNBC_news_data.json"
    click notebook1 "https://github.com/mildsauce9/stock-data-retrieval/blob/main/load_news.ipynb"
    click notebook2 "https://github.com/mildsauce9/stock-data-retrieval/blob/main/view_data.ipynb"
    click cleaned1 "https://github.com/mildsauce9/stock-data-retrieval/blob/main/Cleaned_Reuters_news_data.json"
    click cleaned2 "https://github.com/mildsauce9/stock-data-retrieval/blob/main/Cleaned_CNBC_news_data.json"
    click embedding_off "https://github.com/mildsauce9/stock-data-retrieval/blob/main/model.py"
    click embed_on "https://github.com/mildsauce9/stock-data-retrieval/blob/main/model.py"
    click web "https://github.com/mildsauce9/stock-data-retrieval/blob/main/app.py"
    click browser "https://github.com/mildsauce9/stock-data-retrieval/blob/main/templates/index.html"
    click top_sum "https://github.com/mildsauce9/stock-data-retrieval/blob/main/top_stock_summaries.json"
    click req "https://github.com/mildsauce9/stock-data-retrieval/blob/main/requirements.txt"
    click readme "https://github.com/mildsauce9/stock-data-retrieval/blob/main/README.md"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef external fill:#90EE90,stroke:#000,stroke-width:1px
    classDef datastore fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef ui fill:#F0E68C,stroke:#000,stroke-width:1px
```
