How minniesmick/comp451-project1 works
This repository is a three-model English TweetEval sentiment-analysis project with an optional local Model Lab. Its main workflow prepares a deduplicated, stratified test split; trains two from-scratch models and fine-tunes DistilBERT; then evaluates and compares predictions. Shared training and metrics utilities support those stages. The Model Lab serves saved results and offers live prediction and tokenization. Some subsystem links are based on README documentation because their implementation was not sampled.
Main entry points: serve.py HTTP server and APIs · train.py Scratch-model training · pipeline.py Test inference · prepare_data.py Prepare datasets
Components 16
Data preparation
- Prepare datasets
tools/prepare_data.py - Training pool
tools/prepare_data.py - Held-out test set
tools/prepare_data.py
Model training
- Custom tokenizers
lib/custom_tokenizers.py - Scratch-model training
src/train.py - DistilBERT fine-tuning
src/fine_tune.py - Training engine
lib/common.py - Saved models
src/train.py
Evaluation and analysis
- Test inference
src/pipeline.py - Metrics and predictions
lib/common.py - Model comparison
tools/compare.py - Evaluation results
src/pipeline.py
Local Model Lab
- HTTP server and APIs
model_lab/serve.py - Interactive viewer
model_lab/ui/app.js
Other components
- Project user
- TweetEval source
Connections 25
- Project user → Prepare datasets: runs preparation
README.md - TweetEval source → Prepare datasets: provides splits
tools/prepare_data.py - Prepare datasets → Training pool: writes training pool
tools/prepare_data.py - Prepare datasets → Held-out test set: writes test set
tools/prepare_data.py - Project user → Scratch-model training: starts training
README.md - Project user → DistilBERT fine-tuning: starts fine-tuning
README.md - Training pool → Scratch-model training: trains on
README.md - Training pool → DistilBERT fine-tuning: fine-tunes on
README.md - Scratch-model training → Custom tokenizers: builds tokenizer
src/train.py - Scratch-model training → Training engine: uses training loop
src/train.py - DistilBERT fine-tuning → Training engine: uses training loop
src/fine_tune.py - Scratch-model training → Saved models: saves models
README.md - DistilBERT fine-tuning → Saved models: saves model
README.md - Project user → Test inference: runs evaluation
README.md - Held-out test set → Test inference: supplies test tweets
src/pipeline.py - Saved models → Test inference: loads model
src/pipeline.py - Test inference → Metrics and predictions: computes metrics
src/pipeline.py - Test inference → Evaluation results: writes predictions
src/pipeline.py - Evaluation results → Model comparison: summarizes results
README.md - Project user → HTTP server and APIs: starts viewer
README.md - HTTP server and APIs → Saved models: loads saved models
model_lab/serve.py - HTTP server and APIs → Evaluation results: serves results
model_lab/serve.py - HTTP server and APIs → Interactive viewer: serves interface
model_lab/serve.py - Interactive viewer → HTTP server and APIs: requests predictions
model_lab/ui/app.js - Interactive viewer → HTTP server and APIs: requests tokenization
model_lab/ui/app.js