# itsthetai/frame_detect_nlp architecture

> Architecture diagram of the GitHub repository itsthetai/frame_detect_nlp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/itsthetai/frame_detect_nlp
- Repository: https://github.com/itsthetai/frame_detect_nlp

## Overview

This repository documents a research workflow for weakly supervised policy-frame classification of German news headlines. Headlines and supplemental corpora are prepared and annotated, annotations are reconciled into training data, and a fine-tuned transformer classifies headlines for benchmark evaluation and applied analysis. The README also describes a BERTopic-based cross-outlet prototype. Most pipeline components are notebooks named in the README but absent from the supplied tree, so directory paths identify those documented subsystems; the sole sampled script only converts CHN JSON files to CSV.

Main entry points: [`a._preprocessing+annotation/4_data_semeval_chn_mfc`](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/4_data_semeval_chn_mfc) (Dataset preparation), [`b._fine_tuning+model_application`](https://github.com/itsthetai/frame_detect_nlp/tree/main/b._fine_tuning%2Bmodel_application) (Frame classifier), [`a._preprocessing+annotation/2_llm-titel-annotation-aller-1400-titel_semeval`](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/2_llm-titel-annotation-aller-1400-titel_semeval) (LLM annotation), [`c._real_data_application/Goldstandard_Tagesschau`](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application/Goldstandard_Tagesschau) (Gold-standard curation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and annotation"]
  node_semeval[("SemEval data")]
  node_annotation["LLM annotation"]
  node_humanstudy["Human pre-study"]
  node_suppdata[("CHN and MFC")]
  node_jsoncsv["CHN converter<br/>[json-to-csv.py]"]
  node_normalization["Dataset preparation"]
  node_trainingdata[("Combined training data")]
end

subgraph group_training["Model training"]
  node_finetune["Transformer fine-tuning"]
  node_classifier["Frame classifier"]
end

subgraph group_evaluation["Evaluation and application"]
  node_tagesschau[("Tagesschau headlines")]
  node_goldstandard["Gold-standard curation"]
  node_evaluation["Benchmark evaluation"]
  node_outlets[("News outlets")]
  node_appliedinference["Applied inference"]
end

subgraph group_analysis["Topic analysis"]
  node_clustering["BERTopic clustering"]
  node_frameanalysis["Frame distributions"]
end

node_researcher(("Researcher"))

node_researcher -.->|"provides headlines"| node_annotation
node_semeval -->|"supplies articles"| node_annotation
node_annotation -->|"provides labels"| node_normalization
node_humanstudy -->|"informs preprocessing"| node_normalization
node_suppdata -.->|"provides CHN JSON"| node_jsoncsv
node_jsoncsv -->|"converts CHN files"| node_normalization
node_suppdata -->|"adds corpora"| node_normalization
node_normalization -->|"combines data"| node_trainingdata
node_trainingdata -->|"trains on"| node_finetune
node_finetune -->|"produces model"| node_classifier
node_tagesschau -->|"provides headlines"| node_goldstandard
node_researcher -->|"annotates headlines"| node_goldstandard
node_goldstandard -->|"supplies benchmark"| node_evaluation
node_classifier -->|"predicts frames"| node_evaluation
node_outlets -->|"supplies headlines"| node_appliedinference
node_classifier -->|"classifies headlines"| node_appliedinference
node_appliedinference -->|"groups headlines"| node_clustering
node_clustering -->|"highlights distributions"| node_frameanalysis

click node_semeval "https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/0_data_startingpoint_semeval23_framedetect"
click node_annotation "https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/2_llm-titel-annotation-aller-1400-titel_semeval"
click node_humanstudy "https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/1_annotationsstudie_pre-study_semeval"
click node_suppdata "https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/3_additional_data_chn%2Bmfc"
click node_jsoncsv "https://github.com/itsthetai/frame_detect_nlp/blob/main/a._preprocessing%2Bannotation/3_additional_data_chn%2Bmfc/3.3_vorbereitung_llm_annotation/chn/chn_to_4_parts/json-to-csv.py"
click node_normalization "https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/4_data_semeval_chn_mfc"
click node_trainingdata "https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/4_data_semeval_chn_mfc/4.2_Datensatzauswertung"
click node_finetune "https://github.com/itsthetai/frame_detect_nlp/tree/main/b._fine_tuning%2Bmodel_application"
click node_classifier "https://github.com/itsthetai/frame_detect_nlp/tree/main/b._fine_tuning%2Bmodel_application"
click node_tagesschau "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application/Goldstandard_Tagesschau"
click node_goldstandard "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application/Goldstandard_Tagesschau"
click node_evaluation "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application"
click node_outlets "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application"
click node_appliedinference "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application"
click node_clustering "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application"
click node_frameanalysis "https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_semeval,node_annotation,node_humanstudy,node_suppdata,node_jsoncsv,node_normalization,node_trainingdata toneBlue
class node_finetune,node_classifier toneAmber
class node_tagesschau,node_goldstandard,node_evaluation,node_outlets,node_appliedinference toneMint
class node_clustering,node_frameanalysis toneRose
class node_researcher toneIndigo
```

## Components

### Data and annotation

- [SemEval data](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/0_data_startingpoint_semeval23_framedetect): `a._preprocessing+annotation/0_data_startingpoint_semeval23_framedetect`
- [LLM annotation](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/2_llm-titel-annotation-aller-1400-titel_semeval): `a._preprocessing+annotation/2_llm-titel-annotation-aller-1400-titel_semeval`
- [Human pre-study](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/1_annotationsstudie_pre-study_semeval): `a._preprocessing+annotation/1_annotationsstudie_pre-study_semeval`
- [CHN and MFC](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/3_additional_data_chn%2Bmfc): `a._preprocessing+annotation/3_additional_data_chn+mfc`
- [CHN converter](https://github.com/itsthetai/frame_detect_nlp/blob/main/a._preprocessing%2Bannotation/3_additional_data_chn%2Bmfc/3.3_vorbereitung_llm_annotation/chn/chn_to_4_parts/json-to-csv.py): `a._preprocessing+annotation/3_additional_data_chn+mfc/3.3_vorbereitung_llm_annotation/chn/chn_to_4_parts/json-to-csv.py`
- [Dataset preparation](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/4_data_semeval_chn_mfc): `a._preprocessing+annotation/4_data_semeval_chn_mfc`
- [Combined training data](https://github.com/itsthetai/frame_detect_nlp/tree/main/a._preprocessing%2Bannotation/4_data_semeval_chn_mfc/4.2_Datensatzauswertung): `a._preprocessing+annotation/4_data_semeval_chn_mfc/4.2_Datensatzauswertung`

### Model training

- [Transformer fine-tuning](https://github.com/itsthetai/frame_detect_nlp/tree/main/b._fine_tuning%2Bmodel_application): `b._fine_tuning+model_application`
- [Frame classifier](https://github.com/itsthetai/frame_detect_nlp/tree/main/b._fine_tuning%2Bmodel_application): `b._fine_tuning+model_application`

### Evaluation and application

- [Tagesschau headlines](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application/Goldstandard_Tagesschau): `c._real_data_application/Goldstandard_Tagesschau`
- [Gold-standard curation](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application/Goldstandard_Tagesschau): `c._real_data_application/Goldstandard_Tagesschau`
- [Benchmark evaluation](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application): `c._real_data_application`
- [News outlets](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application): `c._real_data_application`
- [Applied inference](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application): `c._real_data_application`

### Topic analysis

- [BERTopic clustering](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application): `c._real_data_application`
- [Frame distributions](https://github.com/itsthetai/frame_detect_nlp/tree/main/c._real_data_application): `c._real_data_application`

### Other components

- **Researcher**

## Connections

- Researcher → LLM annotation: provides headlines (evidence: `README.md`)
- SemEval data → LLM annotation: supplies articles (evidence: `README.md`)
- LLM annotation → Dataset preparation: provides labels (evidence: `README.md`)
- Human pre-study → Dataset preparation: informs preprocessing (evidence: `README.md`)
- CHN and MFC → CHN converter: provides CHN JSON
- CHN converter → Dataset preparation: converts CHN files (evidence: `a._preprocessing+annotation/3_additional_data_chn+mfc/3.3_vorbereitung_llm_annotation/chn/chn_to_4_parts/json-to-csv.py`)
- CHN and MFC → Dataset preparation: adds corpora (evidence: `README.md`)
- Dataset preparation → Combined training data: combines data (evidence: `README.md`)
- Combined training data → Transformer fine-tuning: trains on (evidence: `README.md`)
- Transformer fine-tuning → Frame classifier: produces model (evidence: `README.md`)
- Tagesschau headlines → Gold-standard curation: provides headlines (evidence: `README.md`)
- Researcher → Gold-standard curation: annotates headlines (evidence: `README.md`)
- Gold-standard curation → Benchmark evaluation: supplies benchmark (evidence: `README.md`)
- Frame classifier → Benchmark evaluation: predicts frames (evidence: `README.md`)
- News outlets → Applied inference: supplies headlines (evidence: `README.md`)
- Frame classifier → Applied inference: classifies headlines (evidence: `README.md`)
- Applied inference → BERTopic clustering: groups headlines (evidence: `README.md`)
- BERTopic clustering → Frame distributions: highlights distributions (evidence: `README.md`)