How 100-hours-a-week/ktb4-whale-ai works
This repository is a collection of independent challenge projects, not a single unified application. The top-level README is just an index; each numbered directory is its own bounded subsystem with its own runtime, dependencies, and workflow.
1. Repo-level structure
- 00_TIL contains small practice scripts for model training/evaluation.
- 01_Weekly_Challenge through 05_Weekly_Challenge are separate weekly deliverables, ranging from CLI tooling to notebooks to an HTTP API.
- 99_Personal_Project is the most complete standalone system: a transformer-based text generation project with tests.
2. 00_TIL: simple ML training/evaluation exercises
- The files preprocess.py, train.py, evaluate.py, and main.py suggest a classic data pipeline split into preprocessing, training, and evaluation entrypoints.
- 01_SLP and 00_SLP_single appear to hold variants of a single-layer perceptron / simple learning pipeline.
- Architecture-wise, this is a lightweight script-based workflow rather than a service: data goes in, transforms happen in preprocessing, model fitting in training, and scoring in evaluation.
3. 01_Weekly_Challenge: file-organizing CLI
- The afo package is a command-line tool.
- Its core is separated into scanner, analyzer, rules, renamer, mover, organizer, and utils, which suggests a staged pipeline.
- Likely flow: scan filesystem contents, analyze file attributes, apply rules, compute new names, move files, and orchestrate the operation from cli.py.
- This is a clear boundary-driven design: filesystem I/O on one side, rule-based organization logic in the middle, and CLI entrypoint on the other.
4. 02_Weekly_Challenge: layered HTTP service
- app/main.py, routers/, crud/, models/, schemas/, and database.py strongly indicate a FastAPI-style CRUD API.
- The routers are split by resource: posts, comments, and summaries.
- Request flow is likely: HTTP request -> router -> CRUD layer -> database -> response schema.
- dummy.json probably acts as seed/sample data or a fixture for local development.
- The presence of ORGANIZING_HTTP.md implies the API was intentionally structured around resource boundaries and layered responsibilities.
5. 03_Weekly_Challenge: notebook-based exploratory work
- This directory is dominated by notebooks for NumPy, Pandas, Matplotlib, dataset collection, and Qiskit.
- There is no application boundary here; it is an analysis and experimentation workspace.
- The architecture is notebook-driven: each notebook is a self-contained environment for a specific topic or experiment.
6. 04_Weekly_Challenge: single notebook challenge
- A single weekly_challenge.ipynb suggests one consolidated analysis or exercise.
- Like 03_Weekly_Challenge, this is exploratory rather than service-oriented, with all logic embedded in notebook cells.
7. 05_Weekly_Challenge: deep learning experiment suite
- This folder contains notebooks for ResNet, VGG16, model comparison, and hyperparameter tuning.
- The structure implies repeated experiment loops: train model variant, compare results, then tune parameters via grid search and random search.
- The architectural boundary here is experimental rather than productized: notebook-driven training/evaluation of computer vision models.
8. 99_Personal_Project: transformer text generation system
- This is the most modular codebase in the repo.
- src/tokenizer/bpe_tokenizer.py handles tokenization.
- src/model/ contains the transformer stack: scaled dot-product attention, multi-head attention, positional encoding, decoder layer, decoder, and full transformer model.
- src/generate.py and src/main.py suggest generation/inference entrypoints.
- Data flow is likely: text input/corpus -> BPE tokenizer -> transformer model -> autoregressive decoding -> generated text output.
9. Testing boundary in the personal project
- The tests/ folder covers tokenizer, attention components, decoder, model, generation, and the full pipeline.
- This indicates the project is organized around verifiable internal layers, not just end-to-end behavior.
- For architecture diagrams, this project should be shown as a stack of reusable model components with tests spanning each layer.
10. Shared operational artifacts
- pyproject.toml, requirements.txt, .env, and dummy.json indicate that some subprojects are meant to be runnable locally with environment configuration and dependency management.
- README.md and RETROSPECTIVE.md files are present throughout, but they are documentation rather than runtime components.
11. How to draw the overall architecture
- Treat the repository as several disconnected systems under one umbrella.
- The main concrete systems worth diagramming are:
- the CLI file organizer in 01_Weekly_Challenge,
- the CRUD HTTP API in 02_Weekly_Challenge,
- the transformer text-generation stack in 99_Personal_Project,
- and the script/notebook-based ML exercises in the other folders.
- There is no evidence of shared runtime coupling between these subprojects; they are independent challenge implementations sharing only the repository.
Main entry points: organizer.py Orchestrator · generate.py Generate · post.py CRUD layer · main.py App entry
Components 23
00_TIL
Script-based ML exercises
- Main script · Top-level exercise entrypoint coordinating data flow.
00_TIL/main.py - Train pipeline · Preprocess, train, and evaluate workflow for the SLP project.
00_TIL/preprocess.py - Perceptron variant · Single-layer perceptron practice implementation.
00_TIL/00_SLP_single/main.py
01_Weekly_Challenge
File-organizing CLI
- CLI entry · User-facing command that launches file organization.
01_Weekly_Challenge/afo/cli.py - Scanner · Collects filesystem contents to be organized.
01_Weekly_Challenge/afo/core/scanner.py - Rules engine · Applies organization rules and file analysis.
01_Weekly_Challenge/afo/core/rules.py - Move flow · Computes names and performs file relocation.
01_Weekly_Challenge/afo/core/renamer.py - Orchestrator · Coordinates scan, analysis, rule application, and movement.
01_Weekly_Challenge/afo/core/organizer.py
02_Weekly_Challenge
Layered HTTP API
- App entry · FastAPI-style application bootstrap and router wiring.
02_Weekly_Challenge/app/main.py - Posts router · HTTP boundary for post resources.
02_Weekly_Challenge/app/routers/posts.py - Comments router · HTTP boundary for comment resources.
02_Weekly_Challenge/app/routers/comments.py - Summaries router · HTTP boundary for summary resources.
02_Weekly_Challenge/app/routers/summaries.py - CRUD layer · Shared create/read/update/delete operations for resources.
02_Weekly_Challenge/app/crud/post.py - Database · Database setup and connection boundary.
02_Weekly_Challenge/app/database.py
03-05 Weekly
Notebook challenge work
- Notebook lab · Self-contained notebook workspace for core data-science topics.
03_Weekly_Challenge/01_NumPy_Quests.ipynb - Single notebook · One consolidated exploratory challenge notebook.
04_Weekly_Challenge/weekly_challenge.ipynb - Vision experiments · Notebook suite for model training, comparison, and tuning.
05_Weekly_Challenge/01_ResNet/01_Challenge.ipynb
99_Personal_Project
Transformer text generation
- Generator entry · Main program for text generation workflow.
99_Personal_Project/src/main.py - Generate · Autoregressive generation and inference orchestration.
99_Personal_Project/src/generate.py - BPE tokenizer · Subword tokenization and detokenization boundary.
99_Personal_Project/src/tokenizer/bpe_tokenizer.py - Attention stack · Scaled dot-product attention and multi-head attention components.
99_Personal_Project/src/model/scaled_dot_product_attention.py - Decoder stack · Positional encoding, decoder layer, decoder, and full model.
99_Personal_Project/src/model/transformer_model.py - Test suite · Layered tests spanning tokenizer, attention, decoder, model, and generation.
99_Personal_Project/tests/test_full_pipeline.py
Connections 21
- Main script → Train pipeline: uses · Entry script drives the preprocessing-training-evaluation workflow.
- Train pipeline → Perceptron variant: variant · A separate perceptron-style implementation sits beside the main pipeline.
- CLI entry → Orchestrator: launches · CLI hands control to the organization workflow.
- Orchestrator → Scanner: scans · Orchestrator starts with filesystem discovery.
- Orchestrator → Rules engine: applies · Rules and analysis determine file handling decisions.
- Orchestrator → Move flow: moves · Organizer delegates renaming and relocation.
- Scanner → Rules engine: feeds · Discovered files are analyzed against the rule set.
- App entry → Posts router: routes · Application exposes post endpoints.
- App entry → Comments router: routes · Application exposes comment endpoints.
- App entry → Summaries router: routes · Application exposes summary endpoints.
- Posts router → CRUD layer: calls · Post endpoints use the shared persistence layer.
- Comments router → CRUD layer: calls · Comment endpoints use the shared persistence layer.
- Summaries router → CRUD layer: calls · Summary endpoints use the shared persistence layer.
- CRUD layer → Database: persists · CRUD operations depend on database setup.
- Generator entry → Generate: invokes · Main program delegates to generation logic.
- Generate → BPE tokenizer: encodes · Text is converted to subword tokens before modeling.
- Generate → Decoder stack: decodes · Generator drives the transformer decoder stack.
- Decoder stack → Attention stack: uses · Decoder internals depend on attention components.
- Decoder stack → Test suite: verified by · Transformer layers are covered by tests.
- BPE tokenizer → Test suite: verified by · Tokenizer behavior is covered by tests.
- Generate → Test suite: verified by · Generation pipeline has dedicated tests.