How itsthetai/frame_detect_nlp works
This repository documents a research workflow for weakly supervised policy-frame classification of German news headlines. Headlines and supplemental corpora are prepared and annotated, annotations are reconciled into training data, and a fine-tuned transformer classifies headlines for benchmark evaluation and applied analysis. The README also describes a BERTopic-based cross-outlet prototype. Most pipeline components are notebooks named in the README but absent from the supplied tree, so directory paths identify those documented subsystems; the sole sampled script only converts CHN JSON files to CSV.
Main entry points: 4_data_semeval_chn_mfc Dataset preparation · b._fine_tuning+model_application Frame classifier · 2_llm-titel-annotation-aller-1400-titel_semeval LLM annotation · Goldstandard_Tagesschau Gold-standard curation
Components 17
Data and annotation
- SemEval data
a._preprocessing+annotation/0_data_startingpoint_semeval23_framedetect - LLM annotation
a._preprocessing+annotation/2_llm-titel-annotation-aller-1400-titel_semeval - Human pre-study
a._preprocessing+annotation/1_annotationsstudie_pre-study_semeval - CHN and MFC
a._preprocessing+annotation/3_additional_data_chn+mfc - CHN converter
a._preprocessing+annotation/3_additional_data_chn+mfc/3.3_vorbereitung_llm_annotation/chn/chn_to_4_parts/json-to-csv.py - Dataset preparation
a._preprocessing+annotation/4_data_semeval_chn_mfc - Combined training data
a._preprocessing+annotation/4_data_semeval_chn_mfc/4.2_Datensatzauswertung
Model training
- Transformer fine-tuning
b._fine_tuning+model_application - Frame classifier
b._fine_tuning+model_application
Evaluation and application
- Tagesschau headlines
c._real_data_application/Goldstandard_Tagesschau - Gold-standard curation
c._real_data_application/Goldstandard_Tagesschau - Benchmark evaluation
c._real_data_application - News outlets
c._real_data_application - Applied inference
c._real_data_application
Topic analysis
- BERTopic clustering
c._real_data_application - Frame distributions
c._real_data_application
Other components
- Researcher
Connections 18
- Researcher → LLM annotation: provides headlines
README.md - SemEval data → LLM annotation: supplies articles
README.md - LLM annotation → Dataset preparation: provides labels
README.md - Human pre-study → Dataset preparation: informs preprocessing
README.md - CHN and MFC → CHN converter: provides CHN JSON
- CHN converter → Dataset preparation: converts CHN files
a._preprocessing+annotation/3_additional_data_chn+mfc/3.3_vorbereitung_llm_annotation/chn/chn_to_4_parts/json-to-csv.py - CHN and MFC → Dataset preparation: adds corpora
README.md - Dataset preparation → Combined training data: combines data
README.md - Combined training data → Transformer fine-tuning: trains on
README.md - Transformer fine-tuning → Frame classifier: produces model
README.md - Tagesschau headlines → Gold-standard curation: provides headlines
README.md - Researcher → Gold-standard curation: annotates headlines
README.md - Gold-standard curation → Benchmark evaluation: supplies benchmark
README.md - Frame classifier → Benchmark evaluation: predicts frames
README.md - News outlets → Applied inference: supplies headlines
README.md - Frame classifier → Applied inference: classifies headlines
README.md - Applied inference → BERTopic clustering: groups headlines
README.md - BERTopic clustering → Frame distributions: highlights distributions
README.md