How theonewhomadethings/craft_gpt works
1. Repository purpose
- Craft_GPT is centered on a text corpus for craft/crochet pattern generation, not on a traditional app backend.
- The repo’s main value is the dataset and the scripts/notebook used to collect, clean, and experiment with generation from that data.
2. Primary data layer
- The patterns/ directory is the dominant asset: hundreds of individual .txt files, each representing a scraped pattern page/article.
- data.txt and data1.txt look like aggregated corpora or training inputs derived from those individual pattern files.
- links.csv appears to be the source index of URLs to scrape.
- labels.txt likely holds class/category labels or metadata for supervised experimentation.
3. Scraping subsystem
- web_scraping/get_webpages.py is the ingestion component that likely reads links.csv, fetches pages, and writes the results into the patterns/ text files.
- web_scraping/test_scrape.py provides a validation boundary for that scraping flow, suggesting the scrape logic is tested separately from generation/training.
4. Text generation / modeling subsystem
- Copy of text_generation.ipynb is the main experimentation environment for model training, prompt engineering, or corpus-based generation.
- This notebook likely consumes the collected pattern texts and the aggregated data*.txt files rather than raw web sources directly.
- The presence of notebook-driven work suggests the project is still research/prototype oriented rather than fully packaged ML service.
5. Application entrypoint
- app.py is the only obvious top-level Python runtime entrypoint outside the scraping and notebook artifacts.
- It likely wraps inference, demo interaction, or a minimal UI/API around the generated text model or dataset.
- Because no dedicated serving framework files are present, this appears to be a lightweight runtime rather than a full service stack.
6. Discord bot sample/integration
- discordbotPUBLIC/ is a self-contained tutorial-style Discord bot project with its own launcher.py, bot package, and extension structure.
- This is architecturally separate from the craft corpus pipeline; it looks like a reusable or experimental integration layer, not the core data pipeline.
- If used, it would act as an interaction surface for the generation system, but it is isolated enough to treat as an auxiliary subsystem.
7. Internal bot structure
- Inside discordbotPUBLIC/Tutorial/bot/, bot.py and extensions/ping.py indicate a modular command-extension pattern typical of Discord bots.
- version.txt and message.txt suggest packaging/bootstrapping metadata for that bot example.
- This subtree is likely independent from the scraping/training assets and should be diagrammed as a separate branch.
8. Main data flow
- External pattern websites feed links.csv.
- The scraper converts those links into local pattern text files in patterns/.
- The local corpus is then consolidated into data.txt / data1.txt and potentially annotated via labels.txt.
- The notebook consumes that prepared corpus to train or experiment with generation.
- app.py and possibly the Discord bot provide user-facing access to the generated content.
9. Key boundaries
- Acquisition boundary: web scraping is isolated under web_scraping/ and should be treated as an ingestion stage.
- Corpus boundary: patterns/ and data*.txt are the canonical local dataset boundary.
- Experiment boundary: the notebook is the modeling sandbox.
- Interaction boundary: app.py and the Discord bot are downstream consumers, not data producers.
10. Architecture takeaway
- This repo is best drawn as a small ML/content-generation pipeline with a large local text corpus, a scraper to build that corpus, a notebook for modeling, and thin runtime surfaces for interaction.
- The Discord bot subtree is a separate example/integration module and should not be merged into the core pipeline in the diagram.
Main entry points: app.py App · patterns Patterns · get_webpages.py Get webpages · Copy of text_generation.ipynb Text notebook
Components 17
Corpus
Local craft-pattern dataset and derived text corpora.
- links.csv · URL list used as scraping input.
links.csv - Patterns · Large collection of scraped pattern text files.
patterns - data.txt · Prepared text corpus derived from scraped patterns.
data.txt - data1.txt · Secondary prepared corpus variant for experiments.
data1.txt - labels.txt · Label or metadata file for corpus experiments.
labels.txt
Scraping
Ingestion flow that collects pattern pages into local files.
- Get webpages · Fetches pages from the link index and writes pattern texts.
web_scraping/get_webpages.py - Scrape test · Validation boundary for the scraping workflow.
web_scraping/test_scrape.py
Modeling
Notebook-based experimentation on the collected corpus.
- Text notebook · Main sandbox for corpus-driven generation experiments.
Copy of text_generation.ipynb
Interaction
Lightweight runtime surfaces for demo or inference.
- App · Lightweight top-level runtime for demo or inference.
app.py
Discord Bot
Standalone example bot subtree separated from the core pipeline.
- Launcher · Starts the standalone Discord bot example.
discordbotPUBLIC/launcher.py - Bot core · Main bot implementation and command handling.
discordbotPUBLIC/Tutorial/bot/bot.py - Ping ext · Example extension module for the bot.
discordbotPUBLIC/Tutorial/bot/extensions/ping.py - Bot package · Package boundary for the tutorial bot code.
discordbotPUBLIC/Tutorial/bot/__init__.py - Version · Bot packaging or release metadata.
discordbotPUBLIC/Tutorial/version.txt - Message · Bot message or bootstrap text asset.
discordbotPUBLIC/message.txt
Connections 19
- links.csv → Get webpages: input · Scraper reads the URL index.
- Get webpages → Patterns: writes · Fetched pages are stored as individual pattern text files.
- Patterns → data.txt: consolidates · Scraped patterns are merged into a training corpus.
- Patterns → data1.txt: consolidates · Alternative corpus assembly from the same pattern set.
- data.txt → Text notebook: consumed by · Notebook experiments use the prepared corpus.
- data1.txt → Text notebook: consumed by · Notebook can compare against a second corpus variant.
- labels.txt → Text notebook: annotates · Labels or metadata support supervised experiments.
- Get webpages → Scrape test: validated by · Scrape flow is checked separately.
- Text notebook → App: informs · Notebook work likely feeds the lightweight runtime.
- App → Patterns: reads corpus · Runtime can reference local pattern content or generated examples.
- App → data.txt: reads corpus · Runtime can use the prepared dataset.
- Launcher → Bot core: starts · Launcher boots the Discord bot core.
- Bot core → Ping ext: loads · Bot core imports extension modules.
- Bot package → Bot core: packages · Package boundary for bot implementation.
- Version → Launcher: metadata · Bot example includes versioning or packaging metadata.
- Message → Launcher: bootstrap text · Bot example uses a local message asset.
- App → Launcher: optional integration · Both are downstream interaction surfaces, but separate runtimes.
- README → Get webpages: documents · Project docs likely describe scraping workflow.
- README → App: documents · Project docs likely mention the runtime entrypoint.