# nlp-with-transformers/notebooks architecture

> Architecture diagram of the GitHub repository nlp-with-transformers/notebooks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nlp-with-transformers/notebooks
- Repository: https://github.com/nlp-with-transformers/notebooks

## Overview

##### Project type and purpose
- This is an open-source companion repository for the O’Reilly book “Natural Language Processing with Transformers.”
- It consists primarily of Jupyter notebooks demonstrating NLP tasks (classification, generation, summarization, etc.) using the Hugging Face Transformers library.
- It provides example code, data samples, environment definitions, utility scripts, and integrations for multiple compute platforms (local, Colab, Kaggle, SageMaker Studio Lab, Paperspace Gradient).

##### File structure
- Top-level “*.ipynb” notebooks (Chapters 01–11): each notebook is a self-contained demo of a transformer task.
- “SageMaker/” folder: notebook variants, training scripts, and helper utilities tailored to AWS SageMaker.
- “data/”: sample data (e.g., `github-issues-transformers.jsonl`) used in demos.
- “scripts/”: small Python scripts (e.g., `create_notebook_table.py`) for repo helper functions.
- “utils.py” & “SageMaker/utils.py”: shared code for data loading, plotting, model wrappers.
- Environment definitions: `environment.yml`, `environment-chapter7.yml`, `requirements*.txt`, and `install.py` for setting up conda/Pip environments.
- GitHub meta: `.github/ISSUE_TEMPLATE`, `.gitignore`, license, README, images.

##### Readme insights
- Describes supported compute platforms (Colab, Kaggle, Gradient, SageMaker Studio Lab).
- Shows that typical usage is “click-to-launch” notebooks in the cloud or clone & conda-install locally.
- Emphasizes GPU requirements, separate env for Chapter 7, and external model/data fetching (internet‐enabled).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and Platform
    User["User / Developer"]:::platform
    subgraph "Platform Options"
        Colab["Google Colab"]:::platform
        Kaggle["Kaggle"]:::platform
        Gradient["Paperspace Gradient"]:::platform
        SageMakerStudioLab["SageMaker Studio Lab"]:::platform
        Local["Local Workstation"]:::platform
    end
    User -->|"launch notebook"| Colab
    User -->|"launch notebook"| Kaggle
    User -->|"launch notebook"| Gradient
    User -->|"launch notebook"| SageMakerStudioLab
    User -->|"clone & setup"| Local

    %% Environment Manager
    subgraph "Environment Manager"
        CondaCore["environment.yml\nrequirements.txt\ninstall.py"]:::code
        CondaChapter7["environment-chapter7.yml\nrequirements-chapter7*.txt"]:::code
    end
    Local -->|"create env"| CondaCore
    Local -->|"create special env"| CondaChapter7
    Colab -->|"uses prebuilt env"| CondaCore
    Kaggle -->|"uses prebuilt env"| CondaCore
    Gradient -->|"uses prebuilt env"| CondaCore
    SageMakerStudioLab -->|"uses prebuilt env"| CondaCore

    %% Jupyter Server
    Jupyter["Jupyter Server"]:::service
    CondaCore -->|"hosts"| Jupyter
    CondaChapter7 -->|"hosts"| Jupyter
    Colab --> Jupyter
    Kaggle --> Jupyter
    Gradient --> Jupyter
    SageMakerStudioLab --> Jupyter
    Local --> Jupyter

    %% Compute Resources
    subgraph "Compute Resources"
        CPU["CPU"]:::compute
        GPU["GPU / CUDA"]:::compute
    end
    Jupyter -->|"allocates resources"| CPU
    Jupyter -->|"allocates resources"| GPU

    %% Notebook Demonstrations
    subgraph "Chapter Notebooks"  
        CN1["01_introduction.ipynb"]:::code
        CN2["02_classification.ipynb"]:::code
        CN3["03_transformer-anatomy.ipynb"]:::code
        CN4["04_multilingual-ner.ipynb"]:::code
        CN5["05_text-generation.ipynb"]:::code
        CN6["06_summarization.ipynb"]:::code
        CN7a["07_question-answering.ipynb"]:::code
        CN7b["07_question_answering_v2.ipynb"]:::code
        CN8["08_model-compression.ipynb"]:::code
        CN9["09_few-to-no-labels.ipynb"]:::code
        CN10["10_transformers-from-scratch.ipynb"]:::code
        CN11["11_future-directions.ipynb"]:::code
    end
    Jupyter -->|"loads"| CN1
    Jupyter --> CN2
    Jupyter --> CN3
    Jupyter --> CN4
    Jupyter --> CN5
    Jupyter --> CN6
    Jupyter --> CN7a
    Jupyter --> CN7b
    Jupyter --> CN8
    Jupyter --> CN9
    Jupyter --> CN10
    Jupyter --> CN11

    %% SageMaker Integration
    subgraph "SageMaker Integration"
        SMN1["SageMaker/01_introduction.ipynb"]:::code
        SMN2["SageMaker/02_classification.ipynb"]:::code
        SMReadme["SageMaker/README.md"]:::code
        SMTrain["SageMaker/scripts/02_classification_train.py"]:::code
        SMUtils["SageMaker/utils.py"]:::code
    end
    SageMakerStudioLab --> SMN1
    SageMakerStudioLab --> SMN2
    SMN1 --> SMUtils
    SMN2 --> SMUtils
    SMN2 --> SMTrain

    %% Shared Utilities & Scripts
    subgraph "Utilities & Scripts"
        Utils["utils.py"]:::code
        MPLStyle["plotting.mplstyle"]:::code
        Scripts["scripts/create_notebook_table.py"]:::code
        OtherScripts["scripts/*"]:::code
    end
    CN1 --> Utils
    CN2 --> Utils
    CN3 --> Utils
    CN4 --> Utils
    CN5 --> Utils
    CN6 --> Utils
    CN7a --> Utils
    CN7b --> Utils
    CN8 --> Utils
    CN9 --> Utils
    CN10 --> Utils
    CN11 --> Utils
    SMN1 --> SMUtils
    SMN2 --> SMUtils
    Scripts -->|"generates index"| CN1
    Utils -->|"styles"| MPLStyle

    %% Data Sources
    subgraph "Data Sources"
        LocalData["data/github-issues-transformers.jsonl"]:::data
        RemoteDataset["Remote datasets (URLs)"]:::data
        HFModelHub["Hugging Face Model Hub"]:::service
    end
    CN* --> LocalData
    CN* --> RemoteDataset
    CN* --> HFModelHub
    SMN* --> HFModelHub

    %% External Services
    subgraph "External Services"
        HuggingFace["Hugging Face APIs"]:::service
        GitHubMeta[".github / README.md / LICENSE"]:::service
    end
    CN* --> HuggingFace
    Scripts --> GitHubMeta
    User --> GitHubMeta

    %% Click Events
    click CN1 "https://github.com/nlp-with-transformers/notebooks/blob/main/01_introduction.ipynb"
    click CN2 "https://github.com/nlp-with-transformers/notebooks/blob/main/02_classification.ipynb"
    click CN3 "https://github.com/nlp-with-transformers/notebooks/blob/main/03_transformer-anatomy.ipynb"
    click CN4 "https://github.com/nlp-with-transformers/notebooks/blob/main/04_multilingual-ner.ipynb"
    click CN5 "https://github.com/nlp-with-transformers/notebooks/blob/main/05_text-generation.ipynb"
    click CN6 "https://github.com/nlp-with-transformers/notebooks/blob/main/06_summarization.ipynb"
    click CN7a "https://github.com/nlp-with-transformers/notebooks/blob/main/07_question-answering.ipynb"
    click CN7b "https://github.com/nlp-with-transformers/notebooks/blob/main/07_question_answering_v2.ipynb"
    click CN8 "https://github.com/nlp-with-transformers/notebooks/blob/main/08_model-compression.ipynb"
    click CN9 "https://github.com/nlp-with-transformers/notebooks/blob/main/09_few-to-no-labels.ipynb"
    click CN10 "https://github.com/nlp-with-transformers/notebooks/blob/main/10_transformers-from-scratch.ipynb"
    click CN11 "https://github.com/nlp-with-transformers/notebooks/blob/main/11_future-directions.ipynb"
    click SMN1 "https://github.com/nlp-with-transformers/notebooks/blob/main/SageMaker/01_introduction.ipynb"
    click SMN2 "https://github.com/nlp-with-transformers/notebooks/blob/main/SageMaker/02_classification.ipynb"
    click SMReadme "https://github.com/nlp-with-transformers/notebooks/blob/main/SageMaker/README.md"
    click SMTrain "https://github.com/nlp-with-transformers/notebooks/blob/main/SageMaker/scripts/02_classification_train.py"
    click SMUtils "https://github.com/nlp-with-transformers/notebooks/blob/main/SageMaker/utils.py"
    click Utils "https://github.com/nlp-with-transformers/notebooks/blob/main/utils.py"
    click MPLStyle "https://github.com/nlp-with-transformers/notebooks/blob/main/plotting.mplstyle"
    click Scripts "https://github.com/nlp-with-transformers/notebooks/blob/main/scripts/create_notebook_table.py"
    click LocalData "https://github.com/nlp-with-transformers/notebooks/blob/main/data/github-issues-transformers.jsonl"

    %% Styles
    classDef code fill:#cfe2f3,stroke:#6fa8dc,color:#073763
    classDef data fill:#d9ead3,stroke:#6aa84f,color:#274e13
    classDef service fill:#d0e0e3,stroke:#8e7cc3,color:#351c75
    classDef compute fill:#fce5cd,stroke:#e69138,color:#7f6000
    classDef platform fill:#ecd9f7,stroke:#a64d79,color:#4c1130
```
