# oyarsa/event_extraction architecture

> Architecture diagram of the GitHub repository oyarsa/event_extraction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oyarsa/event_extraction
- Repository: https://github.com/oyarsa/event_extraction

## Overview

##### Project type and purpose
- Type: A research monorepo with a pipeline for causal event extraction.
- Purpose: Ingest datasets, preprocess them, run multiple extraction models (baseline and LLM‐based, including self‐critique RL), evaluate outputs (automatic and human), and perform error analysis.

##### File structure
- Top-level modules:
– data: raw and processed datasets
– preprocess: conversion scripts (classification, entailment, sequence_labeling, etc.)
– sequence_labelling, extractive_qa, gen_qa: baseline models
– chatgpt: LLM‐based extraction via OpenAI API
– self_critique: LLM‐based extraction with supervised and RL training
– evaluation: automatic metric computation and config
– agreement: human evaluation agreement calculations
– error_analysis: in‐depth error notebooks and scripts
– annotation: a Streamlit‐based annotation UI and deployment
- Config files: each subproject has its own pyproject.toml, config/*.json or .yaml.
- Scripts: run_*.fish/.py to orchestrate experiments per model.

##### README for additional insights
- The root README categorizes subprojects into “Human Evaluation,” “Baseline,” and “Our RL framework.”
- It calls out dependency management (PDM vs pip-tools), but architecture is pipeline-driven.

1. Data Layer
– Raw datasets (data/raw, data/… )
– Processed datasets (data/processed)
2. Preprocessing
– preprocess scripts
3. Baseline Models
– sequence_labelling
– extractive_qa
– gen_qa
4. LLM-Based Extraction
– chatgpt (OpenAI API adapter)
– self_critique (supervised → RL training loop & inference)
5. Evaluation & Metrics
– evaluation module (src/evaluation, scripts, config)
6. Human Evaluation
– agreement (LLM vs human agreement scripts)
7. Error Analysis & Annotation UI
– error_analysis notebooks & scripts
– annotation Streamlit app
– Data Layer → Preprocessing → Models (baseline & LLM-based) → Extracted Outputs
– Extracted Outputs → Evaluation → Metrics
– Extracted Outputs + Human Labels → agreement → agreement statistics
– Extracted Outputs → error_analysis → insights
– self_critique: uses Evaluation as a reward signal to update model (RL loop)
– annotation UI pulls from processed data and pushes annotations back to data layer
– chatgpt and self_critique components call out to OpenAI API (external service)
c. Architectural patterns/design principles:
– Pipeline architecture: clear ETL stages (data → preprocess → model → evaluate).
– Modular subprojects: each model or evaluation variant in its own directory.
– Config-driven experiments: each subproject reads its config/*.json or .yaml.
– Feedback loop: self_critique RL uses evaluation metrics as rewards.
d. Key technologies/frameworks:
– Python scripts & notebooks, Streamlit (annotation UI)
– OpenAI API (chatgpt module)
– PDM / pip-tools for dependency management
– Fish scripts / shell scripts to orchestrate experiments

##### Essential architectural elements
– Show how the self_critique RL framework integrates evaluation to improve the chain of extraction.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        DataRaw["Raw Datasets"]:::dataLayer
        DataProcessed["Processed Datasets"]:::dataLayer
    end

    subgraph "Preprocessing"
        Preprocess["Preprocessing Scripts"]:::preprocessClass
    end

    subgraph "Baseline Models"
        SeqLabel["Sequence Labeling"]:::modelClass
        ExtQA["Extractive QA"]:::modelClass
        GenQA["Generative QA"]:::modelClass
    end

    subgraph "LLM-Based Extraction"
        ChatGPT["ChatGPT Adapter"]:::llmClass
        SelfCritique["Self-Critique RL"]:::llmClass
    end

    ExternalOpenAI["OpenAI API"]:::externalClass

    Extracted["Extracted Outputs"]:::modelClass

    subgraph "Evaluation & Metrics"
        EvalEngine["Evaluation Engine"]:::evalClass
    end

    subgraph "Human Evaluation"
        Agreement["Agreement Scripts"]:::humanClass
    end

    subgraph "Error Analysis & Annotation"
        ErrorAnalysis["Error Analysis"]:::annotationClass
        AnnotationUI["Annotation UI"]:::annotationClass
    end

    DataRaw -->|ingest| Preprocess
    DataProcessed -->|input| Preprocess
    Preprocess -->|preprocessed data| SeqLabel
    Preprocess -->|preprocessed data| ExtQA
    Preprocess -->|preprocessed data| GenQA
    Preprocess -->|preprocessed data| ChatGPT
    Preprocess -->|preprocessed data| SelfCritique

    SeqLabel --> Extracted
    ExtQA --> Extracted
    GenQA --> Extracted
    ChatGPT --> ExternalOpenAI
    ChatGPT --> Extracted
    SelfCritique --> ExternalOpenAI
    SelfCritique --> Extracted

    Extracted -->|model outputs| EvalEngine
    Extracted -->|model outputs| Agreement
    Extracted -->|model outputs| ErrorAnalysis

    EvalEngine -->|reward signal| SelfCritique

    AnnotationUI -->|pull processed data| DataProcessed
    AnnotationUI -->|push annotations| DataRaw

    classDef dataLayer fill:#add8e6,stroke:#333,stroke-width:1px
    classDef preprocessClass fill:#add8e6,stroke:#333,stroke-width:1px
    classDef modelClass fill:#90ee90,stroke:#333,stroke-width:1px,rx:5,ry:5
    classDef llmClass fill:#98fb98,stroke:#333,stroke-width:1px,rx:5,ry:5
    classDef evalClass fill:#ffff99,stroke:#333,stroke-width:1px,shape:hexagon
    classDef humanClass fill:#ffcc99,stroke:#333,stroke-width:1px,shape:cylinder
    classDef annotationClass fill:#ffcc99,stroke:#333,stroke-width:1px,shape:cylinder
    classDef externalClass fill:#d3d3d3,stroke:#333,stroke-width:1px

    click DataRaw "https://github.com/oyarsa/event_extraction/tree/master/data/"
    click DataProcessed "https://github.com/oyarsa/event_extraction/tree/master/data/"
    click Preprocess "https://github.com/oyarsa/event_extraction/tree/master/preprocess/"
    click SeqLabel "https://github.com/oyarsa/event_extraction/tree/master/sequence_labelling/"
    click ExtQA "https://github.com/oyarsa/event_extraction/tree/master/extractive_qa/"
    click GenQA "https://github.com/oyarsa/event_extraction/tree/master/gen_qa/"
    click ChatGPT "https://github.com/oyarsa/event_extraction/tree/master/chatgpt/"
    click SelfCritique "https://github.com/oyarsa/event_extraction/tree/master/self_critique/"
    click EvalEngine "https://github.com/oyarsa/event_extraction/tree/master/evaluation/"
    click Agreement "https://github.com/oyarsa/event_extraction/tree/master/agreement/"
    click ErrorAnalysis "https://github.com/oyarsa/event_extraction/tree/master/error_analysis/"
    click AnnotationUI "https://github.com/oyarsa/event_extraction/tree/master/annotation/"
```
