# sar-hal/bakingllm architecture

> Architecture diagram of the GitHub repository sar-hal/bakingllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sar-hal/bakingllm
- Repository: https://github.com/sar-hal/bakingllm

## Overview

##### Main Project Type and Purpose
- The project is an AI/ML solution with research and production components for natural language processing (NER for recipes). It leverages fine-tuned transformer models (e.g., DistilBERT) to extract ingredient data from varied text formats.
- The overall purpose is to enable precise conversion of vague measurements (like “cups” or “spoons”) into accurate gram measurements for precision baking.

##### File Structure for Architectural Clues
- Two major directories indicate separate model experiments and training pipelines: “DevErrors” and “recipe_ner_model”. Each contains fine-tuned model checkpoints, configuration files, and training logs.
- Multiple Jupyter Notebooks (e.g., train.ipynb, ver2bert.ipynb, no_ner.ipynb) and Python scripts (Extract_Ner.py, Extract_Ner_API.py) show that the project has an exploratory/training phase and an inference/API layer.
- The presence of API scripts (Extract_Ner_API.py) and a hello-world example (helloworld.py) suggest that once trained, models are integrated into an API service to perform NER and conversion tasks.

##### System Components
a. Data Ingestion & Preprocessing:
- Component: Data Sources (raw recipe text, taste sets in CSV/JSONL formats)
- Process: Clean, preprocess, and prepare the input data (as seen in notebooks like cleandata.ipynb and tasteset.csv/tasteset_final.jsonl).
b. Model Training & Experimentation:
- Component(s): Training Pipelines under “DevErrors” and “recipe_ner_model”
- Included items: Notebooks (train.ipynb, trainv2.ipynb, ver2bert.ipynb, etc.), fine-tuned models (DistilBERT variants), checkpoint directories and configuration files.
- Pattern: Iterate on model improvements using different experimental versions and checkpoints.
c. Named Entity Recognition (NER) Extraction Module:
- Component: NER Extraction Scripts and Notebooks (Extract_Ner.ipynb, Extract_Ner.py)
- Role: Process recipe texts to identify ingredients and measurements.
d. Inference & API Layer:
- Component: API Service (Extract_Ner_API.py) and supporting demo scripts (helloworld.py)
- Function: Provide real-time extraction and conversion for recipe data.
e. Optional/Supporting Components:
- Experiment Tracking & Logging: Folder “results” containing training logs and checkpoints.
- Additional utilities: Files like geminiapifetch.ipynb for data fetching, which may connect to external recipe sources or databases.

##### Establish the Relationships and Data Flows
- How outputs of training (model checkpoints, configuration files) feed into the Inference/NER module.
- NER Extraction module to the API layer, which then outputs processed ingredient data.
- Optionally, indicate the integration of a conversion algorithm (even if not explicitly coded, this is implied by the objective to “convert measurements into grams”) which uses NER output along with a (possibly external) database of ingredient densities.

##### Architectural Patterns and Technical Stack
- Modular separation between data preparation, model training, NER extraction, and API service.
- Use of transformer-based architectures (DistilBERT fine tuning) and beacon signs of PyTorch (e.g., rng_state.pth, training_args.bin) plus configuration files suggesting HuggingFace toolkit implementations.
- The iterative experimentation model (multiple versioned checkpoints and notebooks) suggests a research-first approach that later feeds into deployment.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion & Preprocessing Subgraph
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        DA("Raw Recipe Data"):::pre
        CD["Clean Data Notebook"]:::pre
        CSV["Taste Set CSV"]:::pre
        JSONL["Taste Set Final JSONL"]:::pre
    end

    %% Model Training & Experimentation Subgraph
    subgraph "Model Training & Experimentation"
        direction TB
        subgraph "DevErrors Training"
            direction TB
            T1["Train Notebook"]:::training
            T2["Train v2 Notebook"]:::training
            T3["ver2bert Notebook"]:::training
            T4["ver3bert Notebook"]:::training
            T5["ver4bert Notebook"]:::training
        end
        subgraph "Recipe NER Model"
            direction TB
            CFG["Config File"]:::training
            CP["Model Checkpoint"]:::training
        end
    end

    %% Experiment Tracking & Logging Subgraph
    subgraph "Experiment Tracking & Logging"
        direction TB
        LOG1["Training Logs"]:::tracking
        LOG2["Run Logs"]:::tracking
    end

    %% NER Extraction Module Subgraph
    subgraph "NER Extraction Module"
        direction TB
        NER_NB["NER Extraction Notebook"]:::ner
        NER_PY["NER Extraction Script"]:::ner
    end

    %% Inference & API Layer Subgraph
    subgraph "Inference & API Layer"
        direction TB
        API["API Service"]:::api
        HW["Hello World Demo"]:::api
    end

    %% Measurement Conversion Module (Optional)
    MC["Measurement Conversion Module"]:::conversion

    %% Additional Utilities Subgraph
    subgraph "Additional Utilities"
        direction TB
        UTIL["Data Fetching Utility"]:::utils
    end

    %% Data Flow Connections
    CD --> T1
    CSV --> T1
    JSONL --> T1
    UTIL --> DA
    T1 --> LOG1
    CFG --> LOG2
    T3 --> NER_NB
    NER_NB --> NER_PY
    NER_PY --> API
    NER_NB --> MC
    MC --> API
    CP --- API
    T5 --> NER_NB
    T2 --- CFG

    %% Style Classes
    classDef pre fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef training fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef ner fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef api fill:#F08080,stroke:#000,stroke-width:2px;
    classDef tracking fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef utils fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef conversion fill:#B0E0E6,stroke:#000,stroke-width:2px;

    %% Click Events for Data Ingestion & Preprocessing
    click CD "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/cleandata.ipynb"
    click CSV "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/tasteset.csv"
    click JSONL "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/tasteset_final.jsonl"

    %% Click Events for Model Training & Experimentation - DevErrors Training
    click T1 "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/train.ipynb"
    click T2 "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/trainv2.ipynb"
    click T3 "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/ver2bert.ipynb"
    click T4 "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/ver3bert.ipynb"
    click T5 "https://github.com/sar-hal/bakingllm/blob/main/DevErrors/ver4bert.ipynb"

    %% Click Events for Model Training & Experimentation - Recipe NER Model
    click CFG "https://github.com/sar-hal/bakingllm/blob/main/recipe_ner_model/config.json"
    click CP "https://github.com/sar-hal/bakingllm/tree/main/recipe_ner_model/checkpoint-246"

    %% Click Events for NER Extraction Module
    click NER_NB "https://github.com/sar-hal/bakingllm/blob/main/Extract_Ner.ipynb"
    click NER_PY "https://github.com/sar-hal/bakingllm/blob/main/Extract_Ner.py"

    %% Click Events for Inference & API Layer
    click API "https://github.com/sar-hal/bakingllm/blob/main/Extract_Ner_API.py"
    click HW "https://github.com/sar-hal/bakingllm/blob/main/helloworld.py"

    %% Click Events for Experiment Tracking & Logging
    click LOG1 "https://github.com/sar-hal/bakingllm/tree/main/DevErrors/results"
    click LOG2 "https://github.com/sar-hal/bakingllm/tree/main/recipe_ner_model/runs"

    %% Click Event for Additional Utilities
    click UTIL "https://github.com/sar-hal/bakingllm/blob/main/geminiapifetch.ipynb"
```
