How oyarsa/event_extraction works
Project type and purpose
• Type: A research monorepo with a pipeline for causal event extraction.
• Purpose: Ingest datasets, preprocess them, run multiple extraction models (baseline and LLM‐based, including self‐critique RL), evaluate outputs (automatic and human), and perform error analysis.
File structure
• Top-level modules:
– data: raw and processed datasets
– preprocess: conversion scripts (classification, entailment, sequence_labeling, etc.)
– sequence_labelling, extractive_qa, gen_qa: baseline models
– chatgpt: LLM‐based extraction via OpenAI API
– self_critique: LLM‐based extraction with supervised and RL training
– evaluation: automatic metric computation and config
– agreement: human evaluation agreement calculations
– error_analysis: in‐depth error notebooks and scripts
– annotation: a Streamlit‐based annotation UI and deployment
• Config files: each subproject has its own pyproject.toml, config/*.json or .yaml.
• Scripts: run_*.fish/.py to orchestrate experiments per model.
README for additional insights
• The root README categorizes subprojects into “Human Evaluation,” “Baseline,” and “Our RL framework.”
• It calls out dependency management (PDM vs pip-tools), but architecture is pipeline-driven.
1. Data Layer
– Raw datasets (data/raw, data/… )
– Processed datasets (data/processed)
2. Preprocessing
– preprocess scripts
3. Baseline Models
– sequence_labelling
– extractive_qa
– gen_qa
4. LLM-Based Extraction
– chatgpt (OpenAI API adapter)
– self_critique (supervised → RL training loop & inference)
5. Evaluation & Metrics
– evaluation module (src/evaluation, scripts, config)
6. Human Evaluation
– agreement (LLM vs human agreement scripts)
7. Error Analysis & Annotation UI
– error_analysis notebooks & scripts
– annotation Streamlit app
– Data Layer → Preprocessing → Models (baseline & LLM-based) → Extracted Outputs
– Extracted Outputs → Evaluation → Metrics
– Extracted Outputs + Human Labels → agreement → agreement statistics
– Extracted Outputs → error_analysis → insights
– self_critique: uses Evaluation as a reward signal to update model (RL loop)
– annotation UI pulls from processed data and pushes annotations back to data layer
– chatgpt and self_critique components call out to OpenAI API (external service)
c. Architectural patterns/design principles:
– Pipeline architecture: clear ETL stages (data → preprocess → model → evaluate).
– Modular subprojects: each model or evaluation variant in its own directory.
– Config-driven experiments: each subproject reads its config/*.json or .yaml.
– Feedback loop: self_critique RL uses evaluation metrics as rewards.
d. Key technologies/frameworks:
– Python scripts & notebooks, Streamlit (annotation UI)
– OpenAI API (chatgpt module)
– PDM / pip-tools for dependency management
– Fish scripts / shell scripts to orchestrate experiments
Essential architectural elements
– Show how the self_critique RL framework integrates evaluation to improve the chain of extraction.