# harshadeep100/financialnewsqatool architecture

> Architecture diagram of the GitHub repository harshadeep100/financialnewsqatool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harshadeep100/financialnewsqatool
- Repository: https://github.com/harshadeep100/financialnewsqatool

## Overview

##### User Interface (Streamlit Frontend)
- Streamlit web app (initiated by main.py).
- That it is the entry point for user interaction.

##### Data Input & Loader Component (URL & Document Upload)
- How users input data, whether by direct URL entry or uploading documents with URLs.

##### Text Processing Module
- Step where the system takes raw article content and processes it (i.e., text splitting and preparation for embedding).

##### Embedding & Indexing Module
- Clearly denote the process where the processed text is converted into embedding vectors using OpenAI’s embedding tools.
- Integration with FAISS for similarity search and efficient indexing.
- That the output (the FAISS index) is stored in a local pickle file (faiss_store_openai.pkl) for subsequent retrieval.

##### Query Handler & LLM Integration
- How these questions are processed by the LLM (ChatGPT) while leveraging the FAISS index for retrieving relevant source URLs.

##### Configuration & External Services
- A configuration component that outlines the use of the .env file where API keys for OpenAI/HuggingFace are managed.
- Also, highlight the dependency on external APIs (for embeddings and LLM queries).

##### Data Persistence
- Storage of the FAISS index (and potentially other artifacts) in a persistent file, which helps reference data across sessions.

- Use clear and descriptive labels for each component (“Streamlit UI”, “URL Loader”, “Text Processor”, “Embedding & FAISS Index”, “LLM Query Handler”, “Configuration/Env”, “Persistent Storage”).
- From User Input → UI → URL Loader → Text Processor → Embedding & FAISS Index.
- Also, from UI → Query Handler → LLM (and then looping back to UI with response).
- Architectural patterns:
- The separation between data ingestion, processing, and output (UI interaction) is key.
- Asynchronous or streaming behavior if applicable (especially with Streamlit and real-time querying).

- Top Layer: User Interaction via Streamlit UI.
- Middle Layer: Backend processing (Data Loader, Text Processing, Embedding/Indexing, LLM Query Handling).
- Bottom Layer: External Services & Configuration (API keys in .env, OpenAI/HuggingFace APIs, and persistent FAISS storage).

This way, the principal software engineer will have a comprehensive view of the components and interactions, enabling a more straightforward evaluation and potential extension or debugging of the application.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    subgraph "User Interaction Layer"
        UI["Streamlit UI"]:::user
    end

    %% Backend Processing Layer
    subgraph "Backend Processing"
        DataLoader["URL Loader"]:::backend
        TextProcessor["Text Processor"]:::backend
        Embedding["Embedding & FAISS Index"]:::backend
        QueryHandler["LLM Query Handler"]:::user
    end

    %% Configuration & External Services Layer
    subgraph "Configuration & External Services"
        Config["Configuration/Env"]:::config
        ExternalAPIs["External APIs (OpenAI/HuggingFace)"]:::external
        Storage["Persistent Storage\n(faiss_store_openai.pkl)"]:::storage
    end

    %% Data Flow Connections
    UI -->|"submits_URL"| DataLoader
    DataLoader -->|"retrieves_content"| TextProcessor
    TextProcessor -->|"prepares_text"| Embedding
    Embedding -->|"stores_index"| Storage

    UI -->|"asks_query"| QueryHandler
    QueryHandler -->|"retrieves_context"| Embedding
    QueryHandler -->|"LLM_request"| ExternalAPIs
    ExternalAPIs -->|"LLM_response"| QueryHandler
    QueryHandler -->|"returns_answer"| UI

    %% Configuration Influence
    Config ---|"configures"| DataLoader
    Config ---|"configures"| QueryHandler
    Config ---|"configures"| ExternalAPIs

    %% Click Events (Paths not shown in node labels)
    click UI "https://github.com/harshadeep100/financialnewsqatool/blob/master/main.py"
    click DataLoader "https://github.com/harshadeep100/financialnewsqatool/blob/master/main.py"
    click TextProcessor "https://github.com/harshadeep100/financialnewsqatool/blob/master/main.py"
    click Embedding "https://github.com/harshadeep100/financialnewsqatool/blob/master/main.py"
    click QueryHandler "https://github.com/harshadeep100/financialnewsqatool/blob/master/main.py"
    click Config "https://github.com/harshadeep100/financialnewsqatool/blob/master/.env"
    click ExternalAPIs "https://github.com/harshadeep100/financialnewsqatool/blob/master/README.md"

    %% Styles
    classDef user fill:#a2d9ce,stroke:#000,stroke-width:2px;
    classDef backend fill:#aed6f1,stroke:#000,stroke-width:2px;
    classDef config fill:#f5b7b1,stroke:#000,stroke-width:2px,font-weight:bold;
    classDef external fill:#f9e79f,stroke:#000,stroke-width:2px;
    classDef storage fill:#d5f5e3,stroke:#000,stroke-width:2px,stroke-dasharray: 5 5;
```
