# regnrok/rag-based-domain-specific-qa architecture

> Architecture diagram of the GitHub repository regnrok/rag-based-domain-specific-qa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/regnrok/rag-based-domain-specific-qa
- Repository: https://github.com/regnrok/rag-based-domain-specific-qa

## Overview

##### Project type and purpose
- This is a full‑stack, domain‑specific QA web application using a Retrieval‑Augmented Generation (RAG) architecture.
- Purpose: Load financial documents (PDFs), index them in a vector database, retrieve relevant passages at query time, and generate answers with a fine‑tuned Flan‑T5‑XL model. It also supports training, evaluation, and model deployment.

##### File structure
- Top‑level scripts:
- app.py      → Streamlit front‑end
- rag.py      → RAG pipeline (retriever + generator)
- model.py    → Fine‑tuning Flan‑T5‑XL on TAT‑QA
- evaluation.py, testing.py → Model evaluation and tests
- hf_push.py  → Push model to Hugging Face Hub
- Data directories:
- Data/TAT‑QA/dataset_raw → Raw JSON splits for training/eval
- chroma_db               → ChromaDB SQLite files storing embeddings
- output                  → CSV and visualization outputs for evaluation
- Config files:
- requirements.txt        → Dependencies
- Patterns:
- Single‑process Python scripts (no microservices)
- Monolithic repo with clear separation of concerns (UI, pipeline, training, evaluation, deployment)

##### README for additional insights
- Architecture: RAG (retriever + generator) over a vector database (ChromaDB)
- Tech stack:
- Streamlit for UI
- Hugging Face Transformers (Flan‑T5‑XL)
- ChromaDB for vector storage
- Standard ML evaluation metrics (ROUGE, BLEU, F1, etc.)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        Browser["Browser (User)"]:::ui
        StreamlitUI["Streamlit UI\n(app.py)"]:::ui
    end

    Browser -->|HTTP/WS| StreamlitUI

    %% Ingestion & RAG Pipeline
    subgraph "Ingestion & RAG Pipeline"
        PDFProcessor["PDF Processor & Embedder\n(rag.py)"]:::process
        VectorStore["ChromaDB Vector Store"]:::db
        Retriever["Retriever\n(rag.py)"]:::ml
        Generator["Flan‑T5‑XL Generator\n(rag.py)"]:::ml
    end

    StreamlitUI -->|Upload PDF| PDFProcessor
    PDFProcessor -->|"embeddings"| VectorStore
    StreamlitUI -->|Ask question| Retriever
    Retriever -->|search| VectorStore
    Retriever -->|context| Generator
    Generator -->|answer| StreamlitUI

    %% Training & Evaluation
    subgraph "Training & Evaluation"
        RawData["Raw Dataset\n(Data/TAT-QA/dataset_raw/)"]:::data
        Trainer["Model Trainer\n(model.py)"]:::ml
        Checkpoints["Model Checkpoints"]:::db
        Evaluator["Evaluator\n(evaluation.py)"]:::process
        Tester["Testing Suite\n(testing.py)"]:::process
        Outputs["Eval Outputs\n(output/)"]:::db
        HFPush["Hugging Face Push\n(hf_push.py)"]:::process
    end

    RawData --> Trainer
    Trainer --> Checkpoints
    Checkpoints --> Evaluator
    Checkpoints --> Tester
    Evaluator --> Outputs
    Tester --> Outputs
    Outputs --> HFPush
    Trainer --> HFPush

    %% Project Artifacts
    subgraph "Project Artifacts"
        Requirements["requirements.txt"]:::metadata
        Readme["README.md"]:::metadata
        License["LICENSE"]:::metadata
    end

    %% Click Events
    click StreamlitUI "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/app.py"
    click PDFProcessor "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/rag.py"
    click Retriever "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/rag.py"
    click Generator "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/rag.py"
    click VectorStore "https://github.com/regnrok/rag-based-domain-specific-qa/tree/main/chroma_db/"
    click Trainer "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/model.py"
    click Evaluator "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/evaluation.py"
    click Tester "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/testing.py"
    click HFPush "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/hf_push.py"
    click RawData "https://github.com/regnrok/rag-based-domain-specific-qa/tree/main/Data/TAT-QA/dataset_raw/"
    click Outputs "https://github.com/regnrok/rag-based-domain-specific-qa/tree/main/output/"
    click Requirements "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/requirements.txt"
    click Readme "https://github.com/regnrok/rag-based-domain-specific-qa/blob/main/README.md"
    click License "https://github.com/regnrok/rag-based-domain-specific-qa/tree/main/LICENSE"

    %% Styles
    classDef ui fill:#8ecae6,stroke:#023047,color:#000;
    classDef process fill:#ffb703,stroke:#fb8500,color:#000;
    classDef db fill:#219ebc,stroke:#023047,color:#fff;
    classDef ml fill:#fb8500,stroke:#f8961e,color:#000;
    classDef data fill:#8ecae6,stroke:#219ebc,color:#000;
    classDef metadata fill:#cccccc,stroke:#999999,color:#000;
```
