How orionw/report_gen_eval works
Key Components to Include:
Core Evaluation Pipeline
• Use a horizontal flow from left to right showing the evaluation stages
• Three main evaluation branches (sentences with citations, without citations, and ignored statements)
Input Processing Layer
• On the left side
• JSONL input handling
• Data structure transformation
LLM Integration Layer
• In the middle
• Connection to Together AI API
• Different prompt types from the prompts directory
Scoring and Metrics Layer
• On the right side
• Calculation of Precision and Recall metrics
• Aggregation of rewards/penalties
Specific Instructions for Drawing:
• [Input Layer] -> [Evaluation Pipeline] -> [Scoring Layer]
• ↑
• [LLM Integration]
Detail the Components
a. Input Layer (Left):
• Sub-components:
• JSONL Parser
• Citation Mapper
• Sentence Extractor
b. Evaluation Pipeline (Center):
• Citation Evaluation Path
• Non-Citation Path
• Ignore Path
• Decision points with diamonds
c. LLM Integration (Center-Bottom):
• Check Negative
• Check Relevance
• First Instance
• Nugget Agreement
• Requires Citation
• API connection point
d. Scoring Layer (Right):
• Reward Counter
• Penalty Counter
• Metric Calculator
Technical Details
• Together AI API integration point
• Where environment variables are used
• Async operations
Metrics
• How Precision and Recall are calculated
• Where rewards/penalties are tracked
Special Considerations: