# gokkul-ks/langaithon_thamizhnlpers architecture

> Architecture diagram of the GitHub repository gokkul-ks/langaithon_thamizhnlpers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gokkul-ks/langaithon_thamizhnlpers
- Repository: https://github.com/gokkul-ks/langaithon_thamizhnlpers

## Overview

LangAIthon is a Tamil part-of-speech tagging product: it prepares tagged data, fine-tunes XLM-RoBERTa, and presents word-level tags through a Streamlit interface. Its principal user workflow is sentence input through the app to model inference and displayed tags; a separate evaluation path measures predictions against held-out data. The graph also preserves the documented UD conversion, synthetic augmentation, dataset merging, and training stages. The app, inference, and merge implementations were not excerpted, so their internal links are limited to documented or otherwise evidenced relationships.

Main entry points: [`src/langaithon/evaluate.py`](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/evaluate.py) (Held-out evaluation), [`src/langaithon/train.py`](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/train.py) (XLM-R training), [`src/langaithon/inference.py`](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/inference.py) (Word-level inference), [`src/langaithon/data_utils.py`](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/data_utils.py) (Tagged data I/O)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Dataset preparation"]
  node_ud_source["UD treebank"]
  node_ud_conversion["UD conversion<br/>[prepare_ud_data.py]"]
  node_synthetic_generation["Synthetic generation"]
  node_interim_data[("Interim datasets")]
  node_dataset_merge["Dataset merge<br/>[merge_datasets.py]"]
  node_processed_data[("Processed splits")]
  node_data_io["Tagged data I/O<br/>[data_utils.py]"]
end

subgraph group_model["Model lifecycle"]
  node_training["XLM-R training<br/>[train.py]"]
  node_configuration["Paths and hyperparameters<br/>[config.py]"]
  node_trained_model["Fine-tuned model<br/>[train.py]"]
end

subgraph group_runtime["Tagging experience"]
  node_app["Streamlit app<br/>[app.py]"]
  node_inference["Word-level inference<br/>[inference.py]"]
  node_alignment["Subword alignment<br/>[alignment.py]"]
  node_tagged_output["Tagged words"]
end

subgraph group_quality["Evaluation"]
  node_evaluation["Held-out evaluation<br/>[evaluate.py]"]
  node_evaluation_report["Metrics report"]
end

node_user(("Tamil text user"))

node_user -->|"enters sentence"| node_app
node_app -.->|"requests tags"| node_inference
node_inference -->|"aligns predictions"| node_alignment
node_inference -.->|"loads model"| node_trained_model
node_app -.->|"displays tags"| node_tagged_output
node_ud_source -->|"provides treebank"| node_ud_conversion
node_ud_conversion -->|"writes UD splits"| node_interim_data
node_synthetic_generation -->|"writes synthetic splits"| node_interim_data
node_ud_conversion -->|"writes via"| node_data_io
node_synthetic_generation -->|"writes via"| node_data_io
node_interim_data -->|"supplies splits"| node_dataset_merge
node_dataset_merge -->|"builds final splits"| node_processed_data
node_processed_data -->|"supplies training data"| node_training
node_training -->|"reads tagged files"| node_data_io
node_training -->|"uses defaults"| node_configuration
node_training -->|"saves weights"| node_trained_model
node_processed_data -->|"provides test split"| node_evaluation
node_evaluation -->|"reads tagged files"| node_data_io
node_evaluation -->|"calls prediction"| node_inference
node_evaluation -->|"writes metrics"| node_evaluation_report
node_evaluation -->|"loads model"| node_trained_model

click node_app "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/app.py"
click node_inference "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/inference.py"
click node_alignment "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/alignment.py"
click node_ud_source "https://github.com/gokkul-ks/langaithon_thamizhnlpers/tree/main/data/raw"
click node_ud_conversion "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/prepare_ud_data.py"
click node_synthetic_generation "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/generate_synthetic_data.py"
click node_interim_data "https://github.com/gokkul-ks/langaithon_thamizhnlpers/tree/main/data/interim"
click node_dataset_merge "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/merge_datasets.py"
click node_processed_data "https://github.com/gokkul-ks/langaithon_thamizhnlpers/tree/main/data/processed"
click node_data_io "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/data_utils.py"
click node_training "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/train.py"
click node_configuration "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/config.py"
click node_trained_model "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/train.py"
click node_evaluation "https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/evaluate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ud_source,node_ud_conversion,node_synthetic_generation,node_interim_data,node_dataset_merge,node_processed_data,node_data_io,node_user toneBlue
class node_training,node_configuration,node_trained_model toneAmber
class node_app,node_inference,node_alignment,node_tagged_output toneMint
class node_evaluation,node_evaluation_report toneRose
```

## Components

### Dataset preparation

- [UD treebank](https://github.com/gokkul-ks/langaithon_thamizhnlpers/tree/main/data/raw): `data/raw`
- [UD conversion](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/prepare_ud_data.py): `src/langaithon/prepare_ud_data.py`
- [Synthetic generation](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/generate_synthetic_data.py): `src/langaithon/generate_synthetic_data.py`
- [Interim datasets](https://github.com/gokkul-ks/langaithon_thamizhnlpers/tree/main/data/interim): `data/interim`
- [Dataset merge](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/merge_datasets.py): `src/langaithon/merge_datasets.py`
- [Processed splits](https://github.com/gokkul-ks/langaithon_thamizhnlpers/tree/main/data/processed): `data/processed`
- [Tagged data I/O](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/data_utils.py): `src/langaithon/data_utils.py`

### Model lifecycle

- [XLM-R training](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/train.py): `src/langaithon/train.py`
- [Paths and hyperparameters](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/config.py): `src/langaithon/config.py`
- [Fine-tuned model](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/train.py): `src/langaithon/train.py`

### Tagging experience

- [Streamlit app](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/app.py): `src/langaithon/app.py`
- [Word-level inference](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/inference.py): `src/langaithon/inference.py`
- [Subword alignment](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/alignment.py): `src/langaithon/alignment.py`
- **Tagged words**

### Evaluation

- [Held-out evaluation](https://github.com/gokkul-ks/langaithon_thamizhnlpers/blob/main/src/langaithon/evaluate.py): `src/langaithon/evaluate.py`
- **Metrics report**

### Other components

- **Tamil text user**

## Connections

- Tamil text user → Streamlit app: enters sentence (evidence: `README.md`)
- Streamlit app → Word-level inference: requests tags
- Word-level inference → Subword alignment: aligns predictions (evidence: `src/langaithon/alignment.py`)
- Word-level inference → Fine-tuned model: loads model
- Streamlit app → Tagged words: displays tags
- UD treebank → UD conversion: provides treebank (evidence: `README.md`)
- UD conversion → Interim datasets: writes UD splits (evidence: `src/langaithon/prepare_ud_data.py`)
- Synthetic generation → Interim datasets: writes synthetic splits (evidence: `src/langaithon/generate_synthetic_data.py`)
- UD conversion → Tagged data I/O: writes via (evidence: `src/langaithon/prepare_ud_data.py`)
- Synthetic generation → Tagged data I/O: writes via (evidence: `src/langaithon/generate_synthetic_data.py`)
- Interim datasets → Dataset merge: supplies splits (evidence: `README.md`)
- Dataset merge → Processed splits: builds final splits (evidence: `README.md`)
- Processed splits → XLM-R training: supplies training data (evidence: `src/langaithon/train.py`)
- XLM-R training → Tagged data I/O: reads tagged files (evidence: `src/langaithon/train.py`)
- XLM-R training → Paths and hyperparameters: uses defaults (evidence: `src/langaithon/train.py`)
- XLM-R training → Fine-tuned model: saves weights (evidence: `README.md`)
- Processed splits → Held-out evaluation: provides test split (evidence: `README.md`)
- Held-out evaluation → Tagged data I/O: reads tagged files (evidence: `src/langaithon/evaluate.py`)
- Held-out evaluation → Word-level inference: calls prediction (evidence: `src/langaithon/evaluate.py`)
- Held-out evaluation → Metrics report: writes metrics (evidence: `src/langaithon/evaluate.py`)
- Held-out evaluation → Fine-tuned model: loads model (evidence: `src/langaithon/evaluate.py`)
