How plageon/htmlrag works
Main Components Identification
a) Data Processing Pipeline:
• HTML Document Input
• Lossless HTML Cleaning Module
• Two-Step Block-Tree-Based HTML Pruning System
• Embedding-based Pruning
• Generative Pruning
b) Core Modules:
• Search Utils (search_utils/)
• HTML Processing (html4rag/)
• Evaluation Utils (evaluate_utils/)
• LLM Integration (llm_modeling/)
Flow Representation
• Data flow from raw HTML to processed context
• Query processing path
• Model interaction flows
• Evaluation feedback loops
Component Details
Key elements to highlight:
a) HTML Processing Pipeline:
• [Raw HTML] → [Lossless Cleaning] → [Block-Tree Construction] → [Embedding Pruning] → [Generative Pruning] → [Final Context]
b) Model Integration:
• Embedding Models for initial pruning
• Generative Models for refined pruning
• Chat Models for final response generation
c) Evaluation Flow:
• Multiple metric calculations (ROUGE, EM, etc.)
• Comparison with baselines
• Performance feedback
Technical Specifications
Technical details such as:
• Model specifications (e.g., Phi-3.8B, Llama-1B)
• Context window sizes
• Key configuration parameters
• Integration points with external services
Supporting Elements
Add:
• Version information
• Key performance metrics
• System requirements