# orionw/report_gen_eval architecture

> Architecture diagram of the GitHub repository orionw/report_gen_eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/orionw/report_gen_eval
- Repository: https://github.com/orionw/report_gen_eval

## Overview

Key Components to Include:

##### Core Evaluation Pipeline
- Use a horizontal flow from left to right showing the evaluation stages
- Three main evaluation branches (sentences with citations, without citations, and ignored statements)

##### Input Processing Layer
- On the left side
- JSONL input handling
- Data structure transformation

##### LLM Integration Layer
- In the middle
- Connection to Together AI API
- Different prompt types from the prompts directory

##### Scoring and Metrics Layer
- On the right side
- Calculation of Precision and Recall metrics
- Aggregation of rewards/penalties

Specific Instructions for Drawing:

- [Input Layer] -> [Evaluation Pipeline] -> [Scoring Layer]
- ↑
- [LLM Integration]

##### Detail the Components

a. Input Layer (Left):
- Sub-components:
- JSONL Parser
- Citation Mapper
- Sentence Extractor

b. Evaluation Pipeline (Center):
- Citation Evaluation Path
- Non-Citation Path
- Ignore Path
- Decision points with diamonds

c. LLM Integration (Center-Bottom):
- Check Negative
- Check Relevance
- First Instance
- Nugget Agreement
- Requires Citation
- API connection point

d. Scoring Layer (Right):
- Reward Counter
- Penalty Counter
- Metric Calculator

##### Technical Details
- Together AI API integration point
- Where environment variables are used
- Async operations

##### Metrics
- How Precision and Recall are calculated
- Where rewards/penalties are tracked

Special Considerations:

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input
        IP[Input Processor]
        JP[JSONL Parser]
        CM[Citation Mapper]
        SE[Sentence Extractor]
        IP --> JP
        JP --> CM
        CM --> SE
    end

    subgraph "Evaluation Pipeline"
        EP[Evaluation Controller]:::core
        CEP[Citation Evaluation Path]:::core
        NCP[Non-Citation Path]:::core
        IGP[Ignore Path]:::core
        
        EP --> CEP
        EP --> NCP
        EP --> IGP
    end

    subgraph "LLM Integration"
        direction TB
        LLM[Together AI API]
        CN[Check Negative]:::llm
        CR[Check Relevance]:::llm
        FI[First Instance]:::llm
        NA[Nugget Agreement]:::llm
        RC[Requires Citation]:::llm
        
        LLM --> CN
        LLM --> CR
        LLM --> FI
        LLM --> NA
        LLM --> RC
    end

    subgraph "Scoring"
        RW[Reward Counter]
        PN[Penalty Counter]
        MC[Metric Calculator]
        
        RW --> MC
        PN --> MC
    end

    SE --> EP
    CEP --> RW
    NCP --> RW
    IGP --> PN
    
    CEP -.-> LLM
    NCP -.-> LLM
    
    classDef core fill:#f9f,stroke:#333
    classDef llm fill:#bbf,stroke:#333
    
    %% Click events for component mapping
    click EP "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/evaluate_report.py"
    click CEP "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/evaluator.py"
    click IP "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/cli.py"
    click JP "https://github.com/orionw/report_gen_eval/blob/master/example_input.jsonl"
    click CM "https://github.com/orionw/report_gen_eval/blob/master/example_nuggets.jsonl"
    click CN "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/prompts/check_negative.py"
    click CR "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/prompts/check_relevance.py"
    click FI "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/prompts/first_instance.py"
    click NA "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/prompts/nugget_agreement.py"
    click RC "https://github.com/orionw/report_gen_eval/blob/master/report_gen_eval/prompts/requires_citation.py"

    %% Legend
    classDef default fill:#fff,stroke:#333
    classDef input fill:#dfd,stroke:#333
    classDef scoring fill:#fdd,stroke:#333
```
