How padas-lab-de/ir-rag-sigir24-persona-rag works
Project Context and Purpose
• This is a research-oriented, retrieval-augmented generation (RAG) framework that integrates user-centric agents. Its goal is to adapt the retrieval and generation process based on users’ interaction and histories.
• The system uses multi-agent techniques and leverages external APIs (for example, OpenAI and optionally others like Ollama or Mixtral) to generate and build responses tailored to user needs.
Main Components to Highlight
a. Data and Dataset Layer:
– The “data” directory holds dataset samples (e.g. for question answering datasets such as “nq”, “triviaqa”, and “webq”).
– This layer should be depicted as the source of raw inputs to the system.
b. Core RAG Modules (Business Logic):
– The “persona_rag” package is the heart of the system. Within it, you have:
• Agents (modules in persona_rag/agents): encapsulating different user-centric agent behaviors.
• Core functions (in persona_rag/core/generate.py): responsible for generating responses.
• Prompts (in persona_rag/prompts): defining and managing the prompt templates.
• Workflows (in persona_rag/workflows): managing the overall orchestration of multi-agent interactions.
– These components should be grouped together to show how they interact and pass control or data among one another.
c. Execution and Pipeline Scripts:
– The “scripts” directory provides the orchestration of tasks such as data pre-processing (“scripts/data_processing”), evaluation (“scripts/evaluation”), and execution (“scripts/execution”). The main entry point is “scripts/main.py.”
– These scripts should be depicted as the orchestrators that trigger the operations inside the core modules in persona_rag.
d. Evaluation and Logging:
– The “logs” directory holds outputs from different experiments and evaluations (organized by dataset and model type). These log files (JSONs, CSVs) represent recording of performance metrics and results.
– The “tests” directory, representing unit and integration tests, should also be shown as supporting quality assurance for the system.
– Additionally, the “metrics” directory (and parts of “scripts/evaluation/metrics.py”) indicate evaluation components.
e. External API & Configuration:
– Environment variables (for API keys) signal the interaction with external services, especially for language generation endpoints (like OpenAI).
– These integrations can be drawn as external blocks that the core generation modules depend on.
Relationships and Data Flow
– Start with raw data (datasets) which flow into the preprocessing components in the “scripts/data_processing” module.
– Preprocessed data is then fed into the core RAG system (specifically the workflows, retrieval processes, and generation via agents).
– The system uses defined prompts and agent logic to generate the answer. At this point, external API calls (via OpenAI or similar) might be depicted as part of the generation pipeline.
– Once generation is complete, results are built into outputs (CSV, JSON), and logged in the “logs” directory.
– Finally, evaluation scripts (in “scripts/evaluation” and “logs/eval.py”) process these outputs, and tests/metrics components verify accuracy.
Architectural Patterns and Design Principles
• Modularization: The separation into “persona_rag” modules versus “scripts” provides a clean division between core logic and orchestration/execution.
• Multi-agent Architecture: Emphasize the use of multiple “agents” (as implemented in persona_rag/agents) interacting with the core workflows.
• Pipeline/Flow-based Design: Highlight the linear data flow from ingestion, through processing, to final evaluation.