# kirimusha/ananke_lang_model architecture

> Architecture diagram of the GitHub repository kirimusha/ananke_lang_model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kirimusha/ananke_lang_model
- Repository: https://github.com/kirimusha/ananke_lang_model

## Overview

ANANKE is a small Transformer language model for text generation and interactive chat, trained on either a cleaned book or an Excel message corpus. The main paths are book-to-training and chat-export-to-Excel-to-training; the chat runner also accepts user input and generates replies. The README documents these capabilities, while the tree and sampled code show their implementations. The chat-export script writes `result.xlsx` in its working directory, whereas the tracked workbook is under `data_processing/`.

Main entry points: [`main_from_chat.py`](https://github.com/kirimusha/ananke_lang_model/blob/main/main_from_chat.py) (Chat training runner), [`main_from_book.py`](https://github.com/kirimusha/ananke_lang_model/blob/main/main_from_book.py) (Book training runner), [`data_processing/from_book.py`](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/from_book.py) (Book extractor), [`data_processing/from_json_to_excel.py`](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/from_json_to_excel.py) (Excel converter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_corpus["Corpus preparation"]
  node_bookfile["FB2 book<br/>[124875312.fb2]"]
  node_bookcleaner["Book extractor<br/>[from_book.py]"]
  node_booktext["Clean book text<br/>[book_clean.txt]"]
  node_export["Chat export"]
  node_excelprep["Excel converter"]
  node_workbook["Tracked workbook<br/>[result.xlsx]"]
  node_excelinput["Training workbook"]
end

subgraph group_language["Language pipeline"]
  node_tokenizer["Token processing<br/>[custom_tokens.py]"]
  node_encoding["Vocabulary encoding<br/>[indexing.py]"]
  node_transformer["Transformer model<br/>[model.py]"]
end

subgraph group_runtime["Training and interaction"]
  node_bookrunner["Book training runner<br/>[main_from_book.py]"]
  node_chatrunner["Chat training runner<br/>[main_from_chat.py]"]
  node_checkpoint["Model checkpoint"]
end

node_user(("Chat user"))
node_reply["Generated reply"]

node_bookfile -->|"provides book"| node_bookcleaner
node_bookcleaner -->|"writes text"| node_booktext
node_export -->|"provides messages"| node_excelprep
node_excelprep -->|"writes workbook"| node_workbook
node_bookrunner -->|"reads text"| node_booktext
node_chatrunner -->|"reads workbook"| node_excelinput
node_excelinput -->|"supplies corpus"| node_chatrunner
node_bookrunner -->|"tokenizes corpus"| node_tokenizer
node_chatrunner -->|"tokenizes corpus"| node_tokenizer
node_bookrunner -->|"encodes tokens"| node_encoding
node_chatrunner -->|"encodes tokens"| node_encoding
node_bookrunner -->|"trains and generates"| node_transformer
node_chatrunner -->|"trains and generates"| node_transformer
node_bookrunner -->|"saves and loads"| node_checkpoint
node_chatrunner -->|"saves and loads"| node_checkpoint
node_user -->|"enters messages"| node_chatrunner
node_chatrunner -->|"prints reply"| node_user
node_chatrunner -->|"generates text"| node_reply

click node_bookfile "https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/124875312.fb2"
click node_bookcleaner "https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/from_book.py"
click node_booktext "https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/book_clean.txt"
click node_excelprep "https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/from_json_to_excel.py"
click node_workbook "https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/result.xlsx"
click node_tokenizer "https://github.com/kirimusha/ananke_lang_model/blob/main/custom_tokens.py"
click node_encoding "https://github.com/kirimusha/ananke_lang_model/blob/main/indexing.py"
click node_transformer "https://github.com/kirimusha/ananke_lang_model/blob/main/model.py"
click node_bookrunner "https://github.com/kirimusha/ananke_lang_model/blob/main/main_from_book.py"
click node_chatrunner "https://github.com/kirimusha/ananke_lang_model/blob/main/main_from_chat.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bookfile,node_bookcleaner,node_booktext,node_export,node_excelprep,node_workbook,node_excelinput,node_user toneBlue
class node_tokenizer,node_encoding,node_transformer toneAmber
class node_bookrunner,node_chatrunner,node_checkpoint toneMint
class node_reply toneIndigo
```

## Components

### Corpus preparation

- [FB2 book](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/124875312.fb2): `data_processing/124875312.fb2`
- [Book extractor](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/from_book.py): `data_processing/from_book.py`
- [Clean book text](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/book_clean.txt): `data_processing/book_clean.txt`
- **Chat export**
- [Excel converter](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/from_json_to_excel.py): `data_processing/from_json_to_excel.py`
- [Tracked workbook](https://github.com/kirimusha/ananke_lang_model/blob/main/data_processing/result.xlsx): `data_processing/result.xlsx`
- **Training workbook**

### Language pipeline

- [Token processing](https://github.com/kirimusha/ananke_lang_model/blob/main/custom_tokens.py): `custom_tokens.py`
- [Vocabulary encoding](https://github.com/kirimusha/ananke_lang_model/blob/main/indexing.py): `indexing.py`
- [Transformer model](https://github.com/kirimusha/ananke_lang_model/blob/main/model.py): `model.py`

### Training and interaction

- [Book training runner](https://github.com/kirimusha/ananke_lang_model/blob/main/main_from_book.py): `main_from_book.py`
- [Chat training runner](https://github.com/kirimusha/ananke_lang_model/blob/main/main_from_chat.py): `main_from_chat.py`
- **Model checkpoint**

### Other components

- **Chat user**
- **Generated reply**

## Connections

- FB2 book → Book extractor: provides book
- Book extractor → Clean book text: writes text
- Chat export → Excel converter: provides messages
- Excel converter → Tracked workbook: writes workbook
- Book training runner → Clean book text: reads text
- Chat training runner → Training workbook: reads workbook
- Training workbook → Chat training runner: supplies corpus
- Book training runner → Token processing: tokenizes corpus
- Chat training runner → Token processing: tokenizes corpus
- Book training runner → Vocabulary encoding: encodes tokens
- Chat training runner → Vocabulary encoding: encodes tokens
- Book training runner → Transformer model: trains and generates
- Chat training runner → Transformer model: trains and generates
- Book training runner → Model checkpoint: saves and loads
- Chat training runner → Model checkpoint: saves and loads
- Chat user → Chat training runner: enters messages
- Chat training runner → Chat user: prints reply
- Chat training runner → Generated reply: generates text