# ayanaussenbayeva/kazru-ner-representation-shift architecture

> Architecture diagram of the GitHub repository ayanaussenbayeva/kazru-ner-representation-shift, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayanaussenbayeva/kazru-ner-representation-shift
- Repository: https://github.com/ayanaussenbayeva/kazru-ner-representation-shift

## Overview

This repository is a research pipeline for training and analyzing multilingual Kazakh–Russian NER on public-transport complaints. The principal workflow loads tokenized train/test data, fine-tunes XLM-RoBERTa, evaluates entity spans, then examines representation geometry and train–test shift; analysis outputs include tables and figures. The README also documents an optional stratified re-split. The notebook is the main implementation, but its source was not sampled, so notebook-internal wiring is based on README descriptions; the statistical-control script’s input dependencies are explicit.

Main entry points: [`notebooks/ner_representation_analysis.ipynb`](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb) (Training and analysis), [`scripts/make_stratified_split.py`](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/make_stratified_split.py) (Stratified split), [`scripts/representation_shift_checks.py`](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/representation_shift_checks.py) (SWD controls), [`figures`](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/tree/main/figures) (Analysis figures)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_stratify["Stratified split"]
  node_split_files["Stratified split files"]
end

subgraph group_model["NER modeling"]
  node_notebook["Training and analysis"]
  node_evaluation["Entity evaluation"]
end

subgraph group_analysis["Representation analysis"]
  node_controls["SWD controls"]
  node_geometry["Error geometry"]
  node_complaint_shift["Complaint-level shift"]
  node_entity_shift["Entity-level shift"]
  node_error_prediction["Error prediction"]
  node_string_overlap["Entity string overlap"]
  node_projections["Exploratory projections"]
end

subgraph group_outputs["Research outputs"]
  node_figures["Analysis figures"]
  node_swd_table["SWD results table"]
end

node_researcher(("Researcher"))
node_complaints[("Complaint data")]
node_encoder["Pretrained XLM-R"]

node_researcher -->|"runs"| node_notebook
node_complaints -->|"loads splits"| node_notebook
node_complaints -->|"provides splits"| node_stratify
node_researcher -.->|"runs optionally"| node_stratify
node_stratify -->|"writes"| node_split_files
node_split_files -.->|"supplies splits"| node_notebook
node_encoder -->|"initializes encoder"| node_notebook
node_notebook -->|"evaluates"| node_evaluation
node_notebook -->|"computes features"| node_geometry
node_notebook -->|"measures shift"| node_complaint_shift
node_notebook -->|"measures shift"| node_entity_shift
node_notebook -->|"tests prediction"| node_error_prediction
node_notebook -->|"checks overlap"| node_string_overlap
node_notebook -->|"creates projections"| node_projections
node_notebook -->|"provides analysis state"| node_controls
node_controls -->|"writes results"| node_swd_table
node_notebook -->|"saves figures"| node_figures

click node_notebook "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_stratify "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/make_stratified_split.py"
click node_controls "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/representation_shift_checks.py"
click node_evaluation "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_geometry "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_complaint_shift "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_entity_shift "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_error_prediction "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_string_overlap "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_projections "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb"
click node_figures "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/tree/main/figures"
click node_swd_table "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/representation_shift_checks.py"
click node_split_files "https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/make_stratified_split.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_stratify,node_split_files toneBlue
class node_notebook,node_evaluation,node_complaints toneAmber
class node_controls,node_geometry,node_complaint_shift,node_entity_shift,node_error_prediction,node_string_overlap,node_projections toneMint
class node_figures,node_swd_table toneRose
class node_researcher,node_encoder toneIndigo
```

## Components

### Data preparation

- [Stratified split](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/make_stratified_split.py): `scripts/make_stratified_split.py`
- [Stratified split files](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/make_stratified_split.py): `scripts/make_stratified_split.py`

### NER modeling

- [Training and analysis](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`
- [Entity evaluation](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`

### Representation analysis

- [SWD controls](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/representation_shift_checks.py): `scripts/representation_shift_checks.py`
- [Error geometry](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`
- [Complaint-level shift](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`
- [Entity-level shift](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`
- [Error prediction](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`
- [Entity string overlap](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`
- [Exploratory projections](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/notebooks/ner_representation_analysis.ipynb): `notebooks/ner_representation_analysis.ipynb`

### Research outputs

- [Analysis figures](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/tree/main/figures): `figures`
- [SWD results table](https://github.com/ayanaussenbayeva/kazru-ner-representation-shift/blob/main/scripts/representation_shift_checks.py): `scripts/representation_shift_checks.py`

### Other components

- **Researcher**
- **Complaint data**
- **Pretrained XLM-R**

## Connections

- Researcher → Training and analysis: runs
- Complaint data → Training and analysis: loads splits
- Complaint data → Stratified split: provides splits
- Researcher → Stratified split: runs optionally
- Stratified split → Stratified split files: writes
- Stratified split files → Training and analysis: supplies splits
- Pretrained XLM-R → Training and analysis: initializes encoder
- Training and analysis → Entity evaluation: evaluates
- Training and analysis → Error geometry: computes features
- Training and analysis → Complaint-level shift: measures shift
- Training and analysis → Entity-level shift: measures shift
- Training and analysis → Error prediction: tests prediction
- Training and analysis → Entity string overlap: checks overlap
- Training and analysis → Exploratory projections: creates projections
- Training and analysis → SWD controls: provides analysis state
- SWD controls → SWD results table: writes results
- Training and analysis → Analysis figures: saves figures