How hmdrez47-star/minigpt works
MiniGPT is a byte-level text-generation product: users provide UTF-8 prompt bytes, the decoder-only Transformer predicts more bytes, and the web/API layer returns generated text. Separate training reads text or medical-comment CSV data, trains the model, and writes a checkpoint consumed by serving or the CLI generator. The README documents additional corpus-cleaning and advanced tokenizer capabilities; excerpts do not show their callers or full wiring, so those modules are represented without speculative links. Operations infrastructure is omitted from this runtime-focused view.
Main entry points: app.py FastAPI service · train_mini.py Training runner · data_pipeline.py Clean and pack · generate_text.py CLI generator
Components 15
Corpus preparation
- Clean and pack
data_pipeline.py - Corpus statistics
data_pipeline.py
Tokenization
- Character and BPE
tokenizer1.py - Production tokenizer
tokenizer2.py - UTF-8 bytes
train_mini.py
Model and training
- Training runner
train_mini.py - Decoder Transformer
mini_gpt.py - Model checkpoint
train_mini.py
Generation access
- FastAPI service
src/serving/app.py - Generation page
src/serving/app.py - CLI generator
generate_text.py - Prometheus metrics
src/serving/app.py
Other components
- Prompt user
- Text corpus
- Medical comments
data/raw/doctor_comments.csv
Connections 17
- Text corpus → Clean and pack: processes
README.md - Medical comments → Training runner: loads comments
train_mini.py - Text corpus → Training runner: loads text
train_mini.py - Clean and pack → Corpus statistics: computes
data_pipeline.py - Training runner → UTF-8 bytes: encodes bytes
train_mini.py - Training runner → Decoder Transformer: trains
train_mini.py - Training runner → Model checkpoint: saves
train_mini.py - Prompt user → FastAPI service: submits prompt
README.md - FastAPI service → Model checkpoint: loads
src/serving/app.py - FastAPI service → Decoder Transformer: generates with
src/serving/app.py - FastAPI service → Generation page: renders
src/serving/app.py - FastAPI service → Prometheus metrics: exposes
src/serving/app.py - Prompt user → CLI generator: requests generation
generate_text.py - CLI generator → Model checkpoint: loads
generate_text.py - CLI generator → Decoder Transformer: generates with
generate_text.py - Production tokenizer → Clean and pack: tokenizes
- Character and BPE → Clean and pack: tokenizes