# theonewhomadethings/craft_gpt architecture

> Architecture diagram of the GitHub repository theonewhomadethings/craft_gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theonewhomadethings/craft_gpt
- Repository: https://github.com/theonewhomadethings/craft_gpt

## Overview

1. **Repository purpose**
   - `Craft_GPT` is centered on a text corpus for craft/crochet pattern generation, not on a traditional app backend.
   - The repo’s main value is the dataset and the scripts/notebook used to collect, clean, and experiment with generation from that data.

2. **Primary data layer**
   - The `patterns/` directory is the dominant asset: hundreds of individual `.txt` files, each representing a scraped pattern page/article.
   - `data.txt` and `data1.txt` look like aggregated corpora or training inputs derived from those individual pattern files.
   - `links.csv` appears to be the source index of URLs to scrape.
   - `labels.txt` likely holds class/category labels or metadata for supervised experimentation.

3. **Scraping subsystem**
   - `web_scraping/get_webpages.py` is the ingestion component that likely reads `links.csv`, fetches pages, and writes the results into the `patterns/` text files.
   - `web_scraping/test_scrape.py` provides a validation boundary for that scraping flow, suggesting the scrape logic is tested separately from generation/training.

4. **Text generation / modeling subsystem**
   - `Copy of text_generation.ipynb` is the main experimentation environment for model training, prompt engineering, or corpus-based generation.
   - This notebook likely consumes the collected pattern texts and the aggregated `data*.txt` files rather than raw web sources directly.
   - The presence of notebook-driven work suggests the project is still research/prototype oriented rather than fully packaged ML service.

5. **Application entrypoint**
   - `app.py` is the only obvious top-level Python runtime entrypoint outside the scraping and notebook artifacts.
   - It likely wraps inference, demo interaction, or a minimal UI/API around the generated text model or dataset.
   - Because no dedicated serving framework files are present, this appears to be a lightweight runtime rather than a full service stack.

6. **Discord bot sample/integration**
   - `discordbotPUBLIC/` is a self-contained tutorial-style Discord bot project with its own `launcher.py`, bot package, and extension structure.
   - This is architecturally separate from the craft corpus pipeline; it looks like a reusable or experimental integration layer, not the core data pipeline.
   - If used, it would act as an interaction surface for the generation system, but it is isolated enough to treat as an auxiliary subsystem.

7. **Internal bot structure**
   - Inside `discordbotPUBLIC/Tutorial/bot/`, `bot.py` and `extensions/ping.py` indicate a modular command-extension pattern typical of Discord bots.
   - `version.txt` and `message.txt` suggest packaging/bootstrapping metadata for that bot example.
   - This subtree is likely independent from the scraping/training assets and should be diagrammed as a separate branch.

8. **Main data flow**
   - External pattern websites feed `links.csv`.
   - The scraper converts those links into local pattern text files in `patterns/`.
   - The local corpus is then consolidated into `data.txt` / `data1.txt` and potentially annotated via `labels.txt`.
   - The notebook consumes that prepared corpus to train or experiment with generation.
   - `app.py` and possibly the Discord bot provide user-facing access to the generated content.

9. **Key boundaries**
   - **Acquisition boundary:** web scraping is isolated under `web_scraping/` and should be treated as an ingestion stage.
   - **Corpus boundary:** `patterns/` and `data*.txt` are the canonical local dataset boundary.
   - **Experiment boundary:** the notebook is the modeling sandbox.
   - **Interaction boundary:** `app.py` and the Discord bot are downstream consumers, not data producers.

10. **Architecture takeaway**
   - This repo is best drawn as a small ML/content-generation pipeline with a large local text corpus, a scraper to build that corpus, a notebook for modeling, and thin runtime surfaces for interaction.
   - The Discord bot subtree is a separate example/integration module and should not be merged into the core pipeline in the diagram.

Main entry points: [`app.py`](https://github.com/theonewhomadethings/craft_gpt/blob/main/app.py) (App), [`patterns`](https://github.com/theonewhomadethings/craft_gpt/tree/main/patterns) (Patterns), [`web_scraping/get_webpages.py`](https://github.com/theonewhomadethings/craft_gpt/blob/main/web_scraping/get_webpages.py) (Get webpages), [`Copy of text_generation.ipynb`](https://github.com/theonewhomadethings/craft_gpt/blob/HEAD/Copy%20of%20text_generation.ipynb) (Text notebook)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_corpus["Corpus"]
  node_links_csv["links.csv<br/>source index<br/>[links.csv]"]
  node_patterns_dir["Patterns<br/>pattern corpus"]
  node_data_txt["data.txt<br/>aggregated corpus<br/>[data.txt]"]
  node_data1_txt["data1.txt<br/>aggregated corpus<br/>[data1.txt]"]
  node_labels_txt["labels.txt<br/>labels/metadata<br/>[labels.txt]"]
end

subgraph group_scraping["Scraping"]
  node_get_webpages_py["Get webpages<br/>scraper<br/>[get_webpages.py]"]
  node_test_scrape_py["Scrape test<br/>scrape check<br/>[test_scrape.py]"]
end

subgraph group_modeling["Modeling"]
  node_notebook["Text notebook<br/>experiment notebook"]
end

subgraph group_interaction["Interaction"]
  node_app_py["App<br/>runtime entrypoint<br/>[app.py]"]
end

subgraph group_discord["Discord Bot"]
  node_launcher_py["Launcher<br/>bot entrypoint<br/>[launcher.py]"]
  node_bot_py["Bot core<br/>discord bot<br/>[bot.py]"]
  node_ping_py["Ping ext<br/>bot extension<br/>[ping.py]"]
  node_bot_init["Bot package<br/>package init<br/>[__init__.py]"]
  node_version_txt["Version<br/>bot metadata<br/>[version.txt]"]
  node_message_txt["Message<br/>bot metadata<br/>[message.txt]"]
end

node_readme["README<br/>project doc<br/>[README.md]"]
node_license["LICENSE<br/>project doc"]

node_links_csv -->|"input"| node_get_webpages_py
node_get_webpages_py -->|"writes"| node_patterns_dir
node_patterns_dir -->|"consolidates"| node_data_txt
node_patterns_dir -->|"consolidates"| node_data1_txt
node_data_txt -->|"consumed by"| node_notebook
node_data1_txt -->|"consumed by"| node_notebook
node_labels_txt -->|"annotates"| node_notebook
node_get_webpages_py -.->|"validated by"| node_test_scrape_py
node_notebook -.->|"informs"| node_app_py
node_app_py -.->|"reads corpus"| node_patterns_dir
node_app_py -.->|"reads corpus"| node_data_txt
node_launcher_py -->|"starts"| node_bot_py
node_bot_py -->|"loads"| node_ping_py
node_bot_init -.->|"packages"| node_bot_py
node_version_txt -.->|"metadata"| node_launcher_py
node_message_txt -.->|"bootstrap text"| node_launcher_py
node_app_py -.->|"optional integration"| node_launcher_py
node_readme -.->|"documents"| node_get_webpages_py
node_readme -.->|"documents"| node_app_py

click node_links_csv "https://github.com/theonewhomadethings/craft_gpt/blob/main/links.csv"
click node_patterns_dir "https://github.com/theonewhomadethings/craft_gpt/tree/main/patterns"
click node_data_txt "https://github.com/theonewhomadethings/craft_gpt/blob/main/data.txt"
click node_data1_txt "https://github.com/theonewhomadethings/craft_gpt/blob/main/data1.txt"
click node_labels_txt "https://github.com/theonewhomadethings/craft_gpt/blob/main/labels.txt"
click node_get_webpages_py "https://github.com/theonewhomadethings/craft_gpt/blob/main/web_scraping/get_webpages.py"
click node_test_scrape_py "https://github.com/theonewhomadethings/craft_gpt/blob/main/web_scraping/test_scrape.py"
click node_notebook "https://github.com/theonewhomadethings/craft_gpt/blob/main/Copy of text_generation.ipynb"
click node_app_py "https://github.com/theonewhomadethings/craft_gpt/blob/main/app.py"
click node_launcher_py "https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/launcher.py"
click node_bot_py "https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/bot/bot.py"
click node_ping_py "https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/bot/extensions/ping.py"
click node_bot_init "https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/bot/__init__.py"
click node_version_txt "https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/version.txt"
click node_message_txt "https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/message.txt"
click node_readme "https://github.com/theonewhomadethings/craft_gpt/blob/main/README.md"
click node_license "https://github.com/theonewhomadethings/craft_gpt/tree/main/LICENSE"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_links_csv,node_patterns_dir,node_data_txt,node_data1_txt,node_labels_txt toneBlue
class node_get_webpages_py,node_test_scrape_py toneAmber
class node_notebook toneMint
class node_app_py toneRose
class node_launcher_py,node_bot_py,node_ping_py,node_bot_init,node_version_txt,node_message_txt toneIndigo
class node_readme,node_license toneNeutral
```

## Components

### Corpus

Local craft-pattern dataset and derived text corpora.

- [links.csv](https://github.com/theonewhomadethings/craft_gpt/blob/main/links.csv): source index · URL list used as scraping input. · `links.csv`
- [Patterns](https://github.com/theonewhomadethings/craft_gpt/tree/main/patterns): pattern corpus · Large collection of scraped pattern text files. · `patterns`
- [data.txt](https://github.com/theonewhomadethings/craft_gpt/blob/main/data.txt): aggregated corpus · Prepared text corpus derived from scraped patterns. · `data.txt`
- [data1.txt](https://github.com/theonewhomadethings/craft_gpt/blob/main/data1.txt): aggregated corpus · Secondary prepared corpus variant for experiments. · `data1.txt`
- [labels.txt](https://github.com/theonewhomadethings/craft_gpt/blob/main/labels.txt): labels/metadata · Label or metadata file for corpus experiments. · `labels.txt`

### Scraping

Ingestion flow that collects pattern pages into local files.

- [Get webpages](https://github.com/theonewhomadethings/craft_gpt/blob/main/web_scraping/get_webpages.py): scraper · Fetches pages from the link index and writes pattern texts. · `web_scraping/get_webpages.py`
- [Scrape test](https://github.com/theonewhomadethings/craft_gpt/blob/main/web_scraping/test_scrape.py): scrape check · Validation boundary for the scraping workflow. · `web_scraping/test_scrape.py`

### Modeling

Notebook-based experimentation on the collected corpus.

- [Text notebook](https://github.com/theonewhomadethings/craft_gpt/blob/HEAD/Copy%20of%20text_generation.ipynb): experiment notebook · Main sandbox for corpus-driven generation experiments. · `Copy of text_generation.ipynb`

### Interaction

Lightweight runtime surfaces for demo or inference.

- [App](https://github.com/theonewhomadethings/craft_gpt/blob/main/app.py): runtime entrypoint · Lightweight top-level runtime for demo or inference. · `app.py`

### Discord Bot

Standalone example bot subtree separated from the core pipeline.

- [Launcher](https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/launcher.py): bot entrypoint · Starts the standalone Discord bot example. · `discordbotPUBLIC/launcher.py`
- [Bot core](https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/bot/bot.py): discord bot · Main bot implementation and command handling. · `discordbotPUBLIC/Tutorial/bot/bot.py`
- [Ping ext](https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/bot/extensions/ping.py): bot extension · Example extension module for the bot. · `discordbotPUBLIC/Tutorial/bot/extensions/ping.py`
- [Bot package](https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/bot/__init__.py): package init · Package boundary for the tutorial bot code. · `discordbotPUBLIC/Tutorial/bot/__init__.py`
- [Version](https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/Tutorial/version.txt): bot metadata · Bot packaging or release metadata. · `discordbotPUBLIC/Tutorial/version.txt`
- [Message](https://github.com/theonewhomadethings/craft_gpt/blob/main/discordbotPUBLIC/message.txt): bot metadata · Bot message or bootstrap text asset. · `discordbotPUBLIC/message.txt`

### Other components

- [README](https://github.com/theonewhomadethings/craft_gpt/blob/main/README.md): project doc · Top-level project overview and usage notes. · `README.md`
- [LICENSE](https://github.com/theonewhomadethings/craft_gpt/tree/main/LICENSE): project doc · Repository license. · `LICENSE`

## Connections

- links.csv → Get webpages: input · Scraper reads the URL index.
- Get webpages → Patterns: writes · Fetched pages are stored as individual pattern text files.
- Patterns → data.txt: consolidates · Scraped patterns are merged into a training corpus.
- Patterns → data1.txt: consolidates · Alternative corpus assembly from the same pattern set.
- data.txt → Text notebook: consumed by · Notebook experiments use the prepared corpus.
- data1.txt → Text notebook: consumed by · Notebook can compare against a second corpus variant.
- labels.txt → Text notebook: annotates · Labels or metadata support supervised experiments.
- Get webpages → Scrape test: validated by · Scrape flow is checked separately.
- Text notebook → App: informs · Notebook work likely feeds the lightweight runtime.
- App → Patterns: reads corpus · Runtime can reference local pattern content or generated examples.
- App → data.txt: reads corpus · Runtime can use the prepared dataset.
- Launcher → Bot core: starts · Launcher boots the Discord bot core.
- Bot core → Ping ext: loads · Bot core imports extension modules.
- Bot package → Bot core: packages · Package boundary for bot implementation.
- Version → Launcher: metadata · Bot example includes versioning or packaging metadata.
- Message → Launcher: bootstrap text · Bot example uses a local message asset.
- App → Launcher: optional integration · Both are downstream interaction surfaces, but separate runtimes.
- README → Get webpages: documents · Project docs likely describe scraping workflow.
- README → App: documents · Project docs likely mention the runtime entrypoint.