How attharva-j/nlp-genai-playground works
Project Type and Purpose
• The repository has two key aspects. First, several notebooks explore sentiment analysis using word embeddings and various RNN-based models for NLP experiments. Second, and more critically, FinRAG is an AI-powered system designed for financial data retrieval and analysis. It processes Excel files, chunks and vectorizes the data using OpenAI’s embeddings, stores these vectors in Pinecone, and finally lets a user interact with the processed financial data through a conversational, Streamlit-based AI chatbot.
File Structure
• The top levels include a FINRAG directory with key files (e.g., FinRAG.py, FinRAG.ipynb) and a RAWDATA directory containing sample financial Excel files. Additionally, there are standalone files such as app.py (likely the Streamlit interface) and excelparser.py (handling file parsing).
• The presence of these files signals a separation between the business logic (parsing, chunking, vectorization, storage) and the presentation/UI (Streamlit app).
• Configuration and experiment files (Jupyter notebooks), which indicate exploratory work in the NLP domain.
Review README for Further Insights
• The README clearly explains FinRAG’s workflow:
• It first “parses” financial data from Excel files.
• It then “chunks” and vectorizes the extracted text using OpenAI’s text-embedding-3-large model.
• The resulting vectors are stored and later retrieved using Pinecone, enabling rapid queries.
• A dedicated conversational UI (built using Streamlit, launched via app.py) allows users to query the processed data in natural language.
• The README also details the intended future extensions and integration with additional data formats or more advanced retrieval techniques.
Main Components and Their Interactions
a. Data Ingestion & File Parsing:
• Component: RAW Financial Data (e.g., NASDAQ-SAIA.xlsx in ./RAWDATA)
• Module: excelparser.py (responsible for reading and extracting data from Excel files)
b. Data Processing – FinRAG Engine:
• Component: FinRAG Processing Module (represented by FinRAG.py and FinRAG.ipynb)
• Tasks: Apply a smart text chunking strategy, and call OpenAI embedding APIs (text-embedding-3-large) to convert data into fixed-size vectors
c. Vector Storage & Retrieval:
• Component: Pinecone Vector Database (external service)
• Interaction: Processed embeddings are stored and later queried during user interactions
d. Conversational AI Interface:
• Component: Streamlit Web App (invoked via app.py)
• Interaction: Provides a user-friendly, chat-based interface that interacts with the processed data by querying Pinecone and incorporating responses via the LLM
e. External Dependencies and API Integrations:
• OpenAI API for embeddings
• Pinecone service for vector storage, indexed with provided keys (keys.yaml to be set up)
Essential Architectural Elements
• Clear component boundaries: Separating data ingestion, processing, external API integration, persistent storage, and the user interface.
• Use of modern design principles: Modularity for independent testing/upgrading of components (like switching out the embedding service or updating the UI).