# karitselmuthu/hugging_face_fine_tuning architecture

> Architecture diagram of the GitHub repository karitselmuthu/hugging_face_fine_tuning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karitselmuthu/hugging_face_fine_tuning
- Repository: https://github.com/karitselmuthu/hugging_face_fine_tuning

## Overview

This is a reusable text fine-tuning toolkit: task configurations and source datasets become prepared prompt/response splits, which are used to train full models or LoRA adapters, evaluate generations, and generate answers from saved runs. Data integrity checks and task-specific metrics support that workflow; audit and label-comparison commands provide additional analysis. The README also documents a separate, earlier horoscope-specific experiment suite, including MLX QLoRA examples. Connections for that suite are documented at the workflow level; its individual implementation paths were not sampled.

Main entry points: [`src/train_task.py`](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/train_task.py) (Train model), [`src/evaluate_task.py`](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/evaluate_task.py) (Evaluate runs), [`src/prepare_task.py`](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/prepare_task.py) (Prepare splits), [`src/infer_task.py`](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/infer_task.py) (Generate answers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Reusable pipeline"]
  node_task_configs["Task configurations"]
  node_preparation["Prepare splits<br/>[prepare_task.py]"]
  node_prepared_data[("Prepared splits")]
  node_shared_core["Task helpers<br/>[task_core.py]"]
  node_training["Train model<br/>[train_task.py]"]
  node_model_runs[("Saved runs")]
  node_inference["Generate answers<br/>[infer_task.py]"]
end

subgraph group_integrity["Data safeguards"]
  node_integrity_helpers["Data integrity<br/>[data_integrity.py]"]
  node_audit["Audit prepared data<br/>[audit_task_data.py]"]
end

subgraph group_analysis["Evaluation and analysis"]
  node_evaluation["Evaluate runs<br/>[evaluate_task.py]"]
  node_metrics["Task metrics<br/>[task_metrics.py]"]
  node_results[("Evaluation results")]
  node_comparison["Compare label runs"]
end

subgraph group_experiments["Horoscope experiments"]
  node_horoscope_suite["Horoscope examples"]
end

node_user(("Task author"))
node_datasets[("Task datasets")]
node_base_models(("Hugging Face models"))

node_user -->|"configures"| node_task_configs
node_datasets -->|"loads data"| node_preparation
node_task_configs -->|"configures"| node_preparation
node_preparation -->|"formats prompts"| node_shared_core
node_preparation -->|"records fingerprints"| node_integrity_helpers
node_preparation -->|"writes splits"| node_prepared_data
node_user -->|"starts training"| node_training
node_task_configs -->|"configures"| node_training
node_training -->|"reads splits"| node_prepared_data
node_training -->|"uses helpers"| node_shared_core
node_training -->|"verifies data"| node_integrity_helpers
node_base_models -->|"provides starting model"| node_training
node_training -->|"saves run"| node_model_runs
node_user -->|"starts evaluation"| node_evaluation
node_evaluation -->|"loads run helpers"| node_shared_core
node_evaluation -->|"reads split"| node_prepared_data
node_evaluation -->|"verifies data"| node_integrity_helpers
node_evaluation -->|"scores generations"| node_metrics
node_evaluation -->|"writes evaluations"| node_results
node_user -->|"provides inputs"| node_inference
node_inference -->|"formats prompt"| node_shared_core
node_inference -->|"loads saved run"| node_model_runs
node_base_models -.->|"backs saved run"| node_inference
node_audit -->|"checks artifacts"| node_integrity_helpers
node_audit -->|"audits splits"| node_prepared_data
node_comparison -.->|"compares evaluations"| node_results
node_user -->|"runs examples"| node_horoscope_suite
node_horoscope_suite -.->|"uses models"| node_base_models

click node_task_configs "https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/tasks"
click node_preparation "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/prepare_task.py"
click node_prepared_data "https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/data"
click node_shared_core "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/task_core.py"
click node_training "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/train_task.py"
click node_model_runs "https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/models"
click node_inference "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/infer_task.py"
click node_evaluation "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/evaluate_task.py"
click node_metrics "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/task_metrics.py"
click node_results "https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/results"
click node_integrity_helpers "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/data_integrity.py"
click node_audit "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/audit_task_data.py"
click node_comparison "https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/compare_label_evaluations.py"
click node_horoscope_suite "https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/src/horoscope_experiments"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task_configs,node_preparation,node_prepared_data,node_shared_core,node_training,node_model_runs,node_inference toneBlue
class node_integrity_helpers,node_audit,node_datasets toneAmber
class node_evaluation,node_metrics,node_results,node_comparison toneMint
class node_horoscope_suite,node_user toneRose
class node_base_models toneIndigo
```

## Components

### Reusable pipeline

- [Task configurations](https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/tasks): `tasks`
- [Prepare splits](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/prepare_task.py): `src/prepare_task.py`
- [Prepared splits](https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/data): `data`
- [Task helpers](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/task_core.py): `src/task_core.py`
- [Train model](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/train_task.py): `src/train_task.py`
- [Saved runs](https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/models): `models`
- [Generate answers](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/infer_task.py): `src/infer_task.py`

### Data safeguards

- [Data integrity](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/data_integrity.py): `src/data_integrity.py`
- [Audit prepared data](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/audit_task_data.py): `src/audit_task_data.py`

### Evaluation and analysis

- [Evaluate runs](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/evaluate_task.py): `src/evaluate_task.py`
- [Task metrics](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/task_metrics.py): `src/task_metrics.py`
- [Evaluation results](https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/results): `results`
- [Compare label runs](https://github.com/karitselmuthu/hugging_face_fine_tuning/blob/main/src/compare_label_evaluations.py): `src/compare_label_evaluations.py`

### Horoscope experiments

- [Horoscope examples](https://github.com/karitselmuthu/hugging_face_fine_tuning/tree/main/src/horoscope_experiments): `src/horoscope_experiments`

### Other components

- **Task author**
- **Task datasets**
- **Hugging Face models**

## Connections

- Task author → Task configurations: configures (evidence: `README.md`)
- Task datasets → Prepare splits: loads data (evidence: `src/prepare_task.py`)
- Task configurations → Prepare splits: configures (evidence: `src/prepare_task.py`)
- Prepare splits → Task helpers: formats prompts (evidence: `src/prepare_task.py`)
- Prepare splits → Data integrity: records fingerprints (evidence: `src/prepare_task.py`)
- Prepare splits → Prepared splits: writes splits (evidence: `README.md`)
- Task author → Train model: starts training (evidence: `README.md`)
- Task configurations → Train model: configures (evidence: `src/train_task.py`)
- Train model → Prepared splits: reads splits (evidence: `src/train_task.py`)
- Train model → Task helpers: uses helpers (evidence: `src/train_task.py`)
- Train model → Data integrity: verifies data (evidence: `src/train_task.py`)
- Hugging Face models → Train model: provides starting model (evidence: `README.md`)
- Train model → Saved runs: saves run (evidence: `README.md`)
- Task author → Evaluate runs: starts evaluation (evidence: `README.md`)
- Evaluate runs → Task helpers: loads run helpers (evidence: `src/evaluate_task.py`)
- Evaluate runs → Prepared splits: reads split (evidence: `src/evaluate_task.py`)
- Evaluate runs → Data integrity: verifies data (evidence: `src/evaluate_task.py`)
- Evaluate runs → Task metrics: scores generations (evidence: `src/evaluate_task.py`)
- Evaluate runs → Evaluation results: writes evaluations (evidence: `README.md`)
- Task author → Generate answers: provides inputs (evidence: `README.md`)
- Generate answers → Task helpers: formats prompt (evidence: `src/infer_task.py`)
- Generate answers → Saved runs: loads saved run (evidence: `README.md`)
- Hugging Face models → Generate answers: backs saved run
- Audit prepared data → Data integrity: checks artifacts (evidence: `src/audit_task_data.py`)
- Audit prepared data → Prepared splits: audits splits (evidence: `README.md`)
- Compare label runs → Evaluation results: compares evaluations
- Task author → Horoscope examples: runs examples (evidence: `README.md`)
- Horoscope examples → Hugging Face models: uses models
