# plageon/htmlrag architecture

> Architecture diagram of the GitHub repository plageon/htmlrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/plageon/htmlrag
- Repository: https://github.com/plageon/htmlrag

## Overview

##### Main Components Identification

a) Data Processing Pipeline:
- HTML Document Input
- Lossless HTML Cleaning Module
- Two-Step Block-Tree-Based HTML Pruning System
- Embedding-based Pruning
- Generative Pruning

b) Core Modules:
- Search Utils (search_utils/)
- HTML Processing (html4rag/)
- Evaluation Utils (evaluate_utils/)
- LLM Integration (llm_modeling/)

##### Flow Representation

- Data flow from raw HTML to processed context
- Query processing path
- Model interaction flows
- Evaluation feedback loops

##### Component Details

Key elements to highlight:
**a) HTML Processing Pipeline:**
- [Raw HTML] → [Lossless Cleaning] → [Block-Tree Construction] → [Embedding Pruning] → [Generative Pruning] → [Final Context]

**b) Model Integration:**
- Embedding Models for initial pruning
- Generative Models for refined pruning
- Chat Models for final response generation

**c) Evaluation Flow:**
- Multiple metric calculations (ROUGE, EM, etc.)
- Comparison with baselines
- Performance feedback

##### Technical Specifications

Technical details such as:
- Model specifications (e.g., Phi-3.8B, Llama-1B)
- Context window sizes
- Key configuration parameters
- Integration points with external services

##### Supporting Elements

Add:
- Version information
- Key performance metrics
- System requirements

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        HTML[Raw HTML Documents]:::input
        Query[User Queries]:::input
        Search[Search Results]:::input
    end

    subgraph "HTML Processing Pipeline"
        Clean["HTML Cleaning"]:::process
        BlockTree["Block-Tree Generation"]:::process
        EmbedPrune["Embedding-based Pruning"]:::process
        GenPrune["Generative Pruning"]:::process
    end

    subgraph "Model Layer"
        subgraph "Embedding Models"
            BGE["BGE Model"]:::model
            E5["E5-Mistral"]:::model
        end
        
        subgraph "Generative Models"
            Phi["Phi-3.8B"]:::model
            Llama1["Llama-1B"]:::model
        end
        
        Chat["Chat Model (Llama-3.1-70B)"]:::model
    end

    subgraph "Search System"
        SearchPipe["Search Pipeline"]:::search
        BingInt["Bing Integration"]:::search
        Extract["Content Extraction"]:::search
        Rerank["Passage Reranking"]:::search
    end

    subgraph "Evaluation Framework"
        BLEU["BLEU Metrics"]:::eval
        ROUGE["ROUGE Metrics"]:::eval
        RAGAS["RAGAS Metrics"]:::eval
        ASQA["ASQA Evaluation"]:::eval
    end

    subgraph "Context Refinement"
        Refiner["Refiner"]:::refine
        KGRefiner["Knowledge Graph Refiner"]:::refine
        Compressor["Context Compressor"]:::refine
    end

    %% Flow Definition
    HTML --> Clean
    Query --> SearchPipe
    Search --> Extract
    Clean --> BlockTree
    BlockTree --> EmbedPrune
    EmbedPrune --> GenPrune
    
    BGE & E5 --> EmbedPrune
    Phi & Llama1 --> GenPrune
    GenPrune --> Refiner
    Refiner --> Chat
    
    SearchPipe --> BingInt
    BingInt --> Extract
    Extract --> Rerank
    
    Chat --> BLEU & ROUGE & RAGAS & ASQA

    %% Styling
    classDef input fill:#a8d1f0
    classDef process fill:#90EE90
    classDef model fill:#FFB347
    classDef search fill:#DDA0DD
    classDef eval fill:#BA55D3
    classDef refine fill:#98FB98

    %% Click Events
    click Clean "https://github.com/plageon/HtmlRAG/blob/main/html4rag/simplify_html.py"
    click BlockTree "https://github.com/plageon/HtmlRAG/blob/main/html4rag/tree_rerank_tree_gen.py"
    click SearchPipe "https://github.com/plageon/HtmlRAG/blob/main/search_utils/search_pipeline_apply.py"
    click BingInt "https://github.com/plageon/HtmlRAG/blob/main/search_utils/bing_search.py"
    click Extract "https://github.com/plageon/HtmlRAG/blob/main/search_utils/extract_web_contents.py"
    click Rerank "https://github.com/plageon/HtmlRAG/blob/main/search_utils/rerank_passage.py"
    click Phi "https://github.com/plageon/HtmlRAG/tree/main/llm_modeling/Phi35/"
    click Llama1 "https://github.com/plageon/HtmlRAG/tree/main/llm_modeling/Llama323B/"
    click BLEU "https://github.com/plageon/HtmlRAG/tree/main/evaluate_utils/bleu/"
    click ROUGE "https://github.com/plageon/HtmlRAG/tree/main/evaluate_utils/rouge/"
    click RAGAS "https://github.com/plageon/HtmlRAG/tree/main/evaluate_utils/ragas/"
    click ASQA "https://github.com/plageon/HtmlRAG/tree/main/evaluate_utils/asqa_eval/"
    click Refiner "https://github.com/plageon/HtmlRAG/blob/main/flashrag_refiner/refiner.py"
    click KGRefiner "https://github.com/plageon/HtmlRAG/blob/main/flashrag_refiner/kg_refiner.py"
    click Compressor "https://github.com/plageon/HtmlRAG/blob/main/flashrag_refiner/selective_context_compressor.py"
```
