How kirimusha/ananke_lang_model works
ANANKE is a small Transformer language model for text generation and interactive chat, trained on either a cleaned book or an Excel message corpus. The main paths are book-to-training and chat-export-to-Excel-to-training; the chat runner also accepts user input and generates replies. The README documents these capabilities, while the tree and sampled code show their implementations. The chat-export script writes result.xlsx in its working directory, whereas the tracked workbook is under data_processing/.
Main entry points: main_from_chat.py Chat training runner · main_from_book.py Book training runner · from_book.py Book extractor · from_json_to_excel.py Excel converter
Components 15
Corpus preparation
- FB2 book
data_processing/124875312.fb2 - Book extractor
data_processing/from_book.py - Clean book text
data_processing/book_clean.txt - Chat export
- Excel converter
data_processing/from_json_to_excel.py - Tracked workbook
data_processing/result.xlsx - Training workbook
Language pipeline
- Token processing
custom_tokens.py - Vocabulary encoding
indexing.py - Transformer model
model.py
Training and interaction
- Book training runner
main_from_book.py - Chat training runner
main_from_chat.py - Model checkpoint
Other components
- Chat user
- Generated reply
Connections 18
- FB2 book → Book extractor: provides book
- Book extractor → Clean book text: writes text
- Chat export → Excel converter: provides messages
- Excel converter → Tracked workbook: writes workbook
- Book training runner → Clean book text: reads text
- Chat training runner → Training workbook: reads workbook
- Training workbook → Chat training runner: supplies corpus
- Book training runner → Token processing: tokenizes corpus
- Chat training runner → Token processing: tokenizes corpus
- Book training runner → Vocabulary encoding: encodes tokens
- Chat training runner → Vocabulary encoding: encodes tokens
- Book training runner → Transformer model: trains and generates
- Chat training runner → Transformer model: trains and generates
- Book training runner → Model checkpoint: saves and loads
- Chat training runner → Model checkpoint: saves and loads
- Chat user → Chat training runner: enters messages
- Chat training runner → Chat user: prints reply
- Chat training runner → Generated reply: generates text