# minniesmick/comp451-project1 architecture

> Architecture diagram of the GitHub repository minniesmick/comp451-project1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minniesmick/comp451-project1
- Repository: https://github.com/minniesmick/comp451-project1

## Overview

This repository is a three-model English TweetEval sentiment-analysis project with an optional local Model Lab. Its main workflow prepares a deduplicated, stratified test split; trains two from-scratch models and fine-tunes DistilBERT; then evaluates and compares predictions. Shared training and metrics utilities support those stages. The Model Lab serves saved results and offers live prediction and tokenization. Some subsystem links are based on README documentation because their implementation was not sampled.

Main entry points: [`model_lab/serve.py`](https://github.com/minniesmick/comp451-project1/blob/main/model_lab/serve.py) (HTTP server and APIs), [`src/train.py`](https://github.com/minniesmick/comp451-project1/blob/main/src/train.py) (Scratch-model training), [`src/pipeline.py`](https://github.com/minniesmick/comp451-project1/blob/main/src/pipeline.py) (Test inference), [`tools/prepare_data.py`](https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py) (Prepare datasets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_prepare["Prepare datasets<br/>[prepare_data.py]"]
  node_dataset1["Training pool<br/>[prepare_data.py]"]
  node_dataset2["Held-out test set<br/>[prepare_data.py]"]
end

subgraph group_models["Model training"]
  node_tokenizers["Custom tokenizers"]
  node_scratch_train["Scratch-model training<br/>[train.py]"]
  node_finetune["DistilBERT fine-tuning<br/>[fine_tune.py]"]
  node_train_engine["Training engine<br/>[common.py]"]
  node_model_artifacts[("Saved models<br/>[train.py]")]
end

subgraph group_evaluation["Evaluation and analysis"]
  node_inference["Test inference<br/>[pipeline.py]"]
  node_metrics["Metrics and predictions<br/>[common.py]"]
  node_compare["Model comparison<br/>[compare.py]"]
  node_result_files["Evaluation results<br/>[pipeline.py]"]
end

subgraph group_lab["Local Model Lab"]
  node_lab_server["HTTP server and APIs<br/>[serve.py]"]
  node_lab_ui["Interactive viewer<br/>[app.js]"]
end

node_operator(("Project user"))
node_tweet_eval["TweetEval source"]

node_operator -->|"runs preparation"| node_prepare
node_tweet_eval -->|"provides splits"| node_prepare
node_prepare -->|"writes training pool"| node_dataset1
node_prepare -->|"writes test set"| node_dataset2
node_operator -->|"starts training"| node_scratch_train
node_operator -->|"starts fine-tuning"| node_finetune
node_dataset1 -->|"trains on"| node_scratch_train
node_dataset1 -->|"fine-tunes on"| node_finetune
node_scratch_train -->|"builds tokenizer"| node_tokenizers
node_scratch_train -->|"uses training loop"| node_train_engine
node_finetune -->|"uses training loop"| node_train_engine
node_scratch_train -->|"saves models"| node_model_artifacts
node_finetune -->|"saves model"| node_model_artifacts
node_operator -->|"runs evaluation"| node_inference
node_dataset2 -->|"supplies test tweets"| node_inference
node_model_artifacts -->|"loads model"| node_inference
node_inference -->|"computes metrics"| node_metrics
node_inference -->|"writes predictions"| node_result_files
node_result_files -->|"summarizes results"| node_compare
node_operator -->|"starts viewer"| node_lab_server
node_lab_server -->|"loads saved models"| node_model_artifacts
node_lab_server -->|"serves results"| node_result_files
node_lab_server -->|"serves interface"| node_lab_ui
node_lab_ui -->|"requests predictions"| node_lab_server
node_lab_ui -->|"requests tokenization"| node_lab_server

click node_prepare "https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py"
click node_dataset1 "https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py"
click node_dataset2 "https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py"
click node_tokenizers "https://github.com/minniesmick/comp451-project1/blob/main/lib/custom_tokenizers.py"
click node_scratch_train "https://github.com/minniesmick/comp451-project1/blob/main/src/train.py"
click node_finetune "https://github.com/minniesmick/comp451-project1/blob/main/src/fine_tune.py"
click node_train_engine "https://github.com/minniesmick/comp451-project1/blob/main/lib/common.py"
click node_model_artifacts "https://github.com/minniesmick/comp451-project1/blob/main/src/train.py"
click node_inference "https://github.com/minniesmick/comp451-project1/blob/main/src/pipeline.py"
click node_metrics "https://github.com/minniesmick/comp451-project1/blob/main/lib/common.py"
click node_compare "https://github.com/minniesmick/comp451-project1/blob/main/tools/compare.py"
click node_lab_server "https://github.com/minniesmick/comp451-project1/blob/main/model_lab/serve.py"
click node_lab_ui "https://github.com/minniesmick/comp451-project1/blob/main/model_lab/ui/app.js"
click node_result_files "https://github.com/minniesmick/comp451-project1/blob/main/src/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prepare,node_dataset1,node_dataset2,node_operator toneBlue
class node_tokenizers,node_scratch_train,node_finetune,node_train_engine,node_model_artifacts toneAmber
class node_inference,node_metrics,node_compare,node_result_files toneMint
class node_lab_server,node_lab_ui toneRose
class node_tweet_eval toneIndigo
```

## Components

### Data preparation

- [Prepare datasets](https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py): `tools/prepare_data.py`
- [Training pool](https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py): `tools/prepare_data.py`
- [Held-out test set](https://github.com/minniesmick/comp451-project1/blob/main/tools/prepare_data.py): `tools/prepare_data.py`

### Model training

- [Custom tokenizers](https://github.com/minniesmick/comp451-project1/blob/main/lib/custom_tokenizers.py): `lib/custom_tokenizers.py`
- [Scratch-model training](https://github.com/minniesmick/comp451-project1/blob/main/src/train.py): `src/train.py`
- [DistilBERT fine-tuning](https://github.com/minniesmick/comp451-project1/blob/main/src/fine_tune.py): `src/fine_tune.py`
- [Training engine](https://github.com/minniesmick/comp451-project1/blob/main/lib/common.py): `lib/common.py`
- [Saved models](https://github.com/minniesmick/comp451-project1/blob/main/src/train.py): `src/train.py`

### Evaluation and analysis

- [Test inference](https://github.com/minniesmick/comp451-project1/blob/main/src/pipeline.py): `src/pipeline.py`
- [Metrics and predictions](https://github.com/minniesmick/comp451-project1/blob/main/lib/common.py): `lib/common.py`
- [Model comparison](https://github.com/minniesmick/comp451-project1/blob/main/tools/compare.py): `tools/compare.py`
- [Evaluation results](https://github.com/minniesmick/comp451-project1/blob/main/src/pipeline.py): `src/pipeline.py`

### Local Model Lab

- [HTTP server and APIs](https://github.com/minniesmick/comp451-project1/blob/main/model_lab/serve.py): `model_lab/serve.py`
- [Interactive viewer](https://github.com/minniesmick/comp451-project1/blob/main/model_lab/ui/app.js): `model_lab/ui/app.js`

### Other components

- **Project user**
- **TweetEval source**

## Connections

- Project user → Prepare datasets: runs preparation (evidence: `README.md`)
- TweetEval source → Prepare datasets: provides splits (evidence: `tools/prepare_data.py`)
- Prepare datasets → Training pool: writes training pool (evidence: `tools/prepare_data.py`)
- Prepare datasets → Held-out test set: writes test set (evidence: `tools/prepare_data.py`)
- Project user → Scratch-model training: starts training (evidence: `README.md`)
- Project user → DistilBERT fine-tuning: starts fine-tuning (evidence: `README.md`)
- Training pool → Scratch-model training: trains on (evidence: `README.md`)
- Training pool → DistilBERT fine-tuning: fine-tunes on (evidence: `README.md`)
- Scratch-model training → Custom tokenizers: builds tokenizer (evidence: `src/train.py`)
- Scratch-model training → Training engine: uses training loop (evidence: `src/train.py`)
- DistilBERT fine-tuning → Training engine: uses training loop (evidence: `src/fine_tune.py`)
- Scratch-model training → Saved models: saves models (evidence: `README.md`)
- DistilBERT fine-tuning → Saved models: saves model (evidence: `README.md`)
- Project user → Test inference: runs evaluation (evidence: `README.md`)
- Held-out test set → Test inference: supplies test tweets (evidence: `src/pipeline.py`)
- Saved models → Test inference: loads model (evidence: `src/pipeline.py`)
- Test inference → Metrics and predictions: computes metrics (evidence: `src/pipeline.py`)
- Test inference → Evaluation results: writes predictions (evidence: `src/pipeline.py`)
- Evaluation results → Model comparison: summarizes results (evidence: `README.md`)
- Project user → HTTP server and APIs: starts viewer (evidence: `README.md`)
- HTTP server and APIs → Saved models: loads saved models (evidence: `model_lab/serve.py`)
- HTTP server and APIs → Evaluation results: serves results (evidence: `model_lab/serve.py`)
- HTTP server and APIs → Interactive viewer: serves interface (evidence: `model_lab/serve.py`)
- Interactive viewer → HTTP server and APIs: requests predictions (evidence: `model_lab/ui/app.js`)
- Interactive viewer → HTTP server and APIs: requests tokenization (evidence: `model_lab/ui/app.js`)
