How cv-lee/financerag works
Project Type and Purpose
• FinanceRAG is a finance-specialized Retrieval-Augmented Generation (RAG) system built for the ACM-ICAIF ’24 Competition. Its main purpose is to retrieve the most relevant financial documents from multiple datasets and then use generative models (via OpenAI API) to produce accurate answers from the retrieved documents.
• The system covers multiple stages: pre-retrieval (query expansion and corpus refinement), multi-stage reranking of retrieved documents, and long-context LLM generation.
Break Down the File Structure
• The top-level directories include:
• “financerag/”: The main module directory containing subdirectories for common utilities, retrieval methods, reranking algorithms, generation code, and task definitions.
• “dataset/”: Storage for multiple finance-related datasets (FinDER, FinQA, etc.).
• Supporting scripts: “pre_retrieval.py”, “prepare_dataset.py”, “rerank.py”, and “run.sh” – these indicate pipeline orchestration and setup.
• Within “financerag/”:
• “common/” holds shared functionality like protocols, loader, and utility functions.
• “retrieval/” includes implementations for BM25, dense retrieval, and sentence encoder-based retrieval – forming the first stage.
• “rerank/” features modules like the cross_encoder which refine and re-rank the initially retrieved candidate documents.
• “generate/” handles the response generation process (likely using LLM endpoints like OpenAI).
• “tasks/” contains modules dedicated to different finance tasks (e.g., FinQATask, ConvFinQATask); these components map to specific datasets and task logic.
Architecture Components and Data Flow
a. Components to Include:
• Pre-Retrieval Module: Triggered by “pre_retrieval.py” and “prepare_dataset.py”, leveraging “prompt.json” for configuration. This module handles query expansion and corpus refinement.
• Retrieval Component: In “financerag/retrieval/” with distinct strategies such as BM25 and dense retrieval. This is where the system initially finds candidate documents.
• Reranking Component: Found in “financerag/rerank/”, which applies multi-stage embedding or cross-encoder based reordering to improve retrieval quality.
• Generation Module: Located in “financerag/generate/”. This part gets final output by interfacing with an LLM (like OpenAI’s API) to generate answers from the retrieved content.
• Task/Orchestration Layer: In “financerag/tasks/” – it maps specific strategies and logic to finance-related tasks (handling different datasets and queries).
• Dataset Management: The “dataset/” directory stores the various finance datasets (e.g., FinDER, FinQA, etc.) each with its preprocessed JSONL queries.
b. Relationships and Interactions:
• The query then flows into the Retrieval component, which uses various retrieval techniques to fetch top candidate documents.
• Retrieved documents are passed into the Reranking component, where they are refined and ordered.
• The best candidates then feed into the Generation module that, leveraging LLM capabilities, forms the final answer.
• The task modules help branch off the workflow based on dataset specifics, enabling specialized processing for each finance dataset.
c. Architectural Patterns and Principles:
• Separation of Concerns: Each main phase (pre-retrieval, retrieval, reranking, generation) is isolated in its dedicated module or package.
• Modular Design: Independent modules (in “common”, “retrieval”, “rerank”, “generate”) promote extensibility, easing individual component development and testing.
• Pipeline Flow: The design follows a sequential data processing pipeline, with modules neatly feeding into one another through well-defined input/output interfaces.
• Configurability: Use of JSON-based prompt files and external environment configuration (via .env) further emphasizes the design’s flexibility for different deployment and testing contexts.
d. Relevant Technologies and Dependencies:
• Python 3.10+ as the primary programming environment.
• CUDA (12.2+) for any GPU-accelerated computations.
• OpenAI and Kaggle APIs for LLM functionality and dataset access.
• Mention any key libraries in requirements.txt that support retrieval (e.g., transformer-based models) or generation (Flash Attention, etc.).
Elements to Include for the Principal Software Engineer
• Use clear labels for each of the following components:
• Dataset Management (datasets folder)
• Pre-Retrieval Engine (scripts: pre_retrieval.py, prompt.json)
• Retrieval Engine (modules within financerag/retrieval: BM25, Dense Retrieval, Sentence Encoder)
• Reranking Engine (modules within financerag/rerank: cross_encoder)
• Generation Engine (modules within financerag/generate, interfacing with OpenAI API)
• Task Management/Orchestration (modules in financerag/tasks)
• Starting from query input (and dataset sourcing) to pre-retrieval.
• From pre-retrieval to retrieval, then into reranking, and finally into generation.
• Data processing components vs. orchestration/logical flow layers.
• Internal modules (common functionalities) vs. externally interfaced services (e.g., OpenAI API, Kaggle integration).
Final Advice
• Adding annotations for each component to explain its responsibilities or key algorithms (e.g., “BM25 for initial candidate retrieval”, “Cross-Encoder for reranking accuracy”).
• This design should serve both as a high-level architectural overview and a detailed guide for how data and control flow through the FinanceRAG system.