How harshadeep100/financialnewsqatool works
User Interface (Streamlit Frontend)
• Streamlit web app (initiated by main.py).
• That it is the entry point for user interaction.
Data Input & Loader Component (URL & Document Upload)
• How users input data, whether by direct URL entry or uploading documents with URLs.
Text Processing Module
• Step where the system takes raw article content and processes it (i.e., text splitting and preparation for embedding).
Embedding & Indexing Module
• Clearly denote the process where the processed text is converted into embedding vectors using OpenAI’s embedding tools.
• Integration with FAISS for similarity search and efficient indexing.
• That the output (the FAISS index) is stored in a local pickle file (faiss_store_openai.pkl) for subsequent retrieval.
Query Handler & LLM Integration
• How these questions are processed by the LLM (ChatGPT) while leveraging the FAISS index for retrieving relevant source URLs.
Configuration & External Services
• A configuration component that outlines the use of the .env file where API keys for OpenAI/HuggingFace are managed.
• Also, highlight the dependency on external APIs (for embeddings and LLM queries).
Data Persistence
• Storage of the FAISS index (and potentially other artifacts) in a persistent file, which helps reference data across sessions.
• Use clear and descriptive labels for each component (“Streamlit UI”, “URL Loader”, “Text Processor”, “Embedding & FAISS Index”, “LLM Query Handler”, “Configuration/Env”, “Persistent Storage”).
• From User Input → UI → URL Loader → Text Processor → Embedding & FAISS Index.
• Also, from UI → Query Handler → LLM (and then looping back to UI with response).
• Architectural patterns:
• The separation between data ingestion, processing, and output (UI interaction) is key.
• Asynchronous or streaming behavior if applicable (especially with Streamlit and real-time querying).
• Top Layer: User Interaction via Streamlit UI.
• Middle Layer: Backend processing (Data Loader, Text Processing, Embedding/Indexing, LLM Query Handling).
• Bottom Layer: External Services & Configuration (API keys in .env, OpenAI/HuggingFace APIs, and persistent FAISS storage).
This way, the principal software engineer will have a comprehensive view of the components and interactions, enabling a more straightforward evaluation and potential extension or debugging of the application.