How sar-hal/bakingllm works
Main Project Type and Purpose
• The project is an AI/ML solution with research and production components for natural language processing (NER for recipes). It leverages fine-tuned transformer models (e.g., DistilBERT) to extract ingredient data from varied text formats.
• The overall purpose is to enable precise conversion of vague measurements (like “cups” or “spoons”) into accurate gram measurements for precision baking.
File Structure for Architectural Clues
• Two major directories indicate separate model experiments and training pipelines: “DevErrors” and “recipe_ner_model”. Each contains fine-tuned model checkpoints, configuration files, and training logs.
• Multiple Jupyter Notebooks (e.g., train.ipynb, ver2bert.ipynb, no_ner.ipynb) and Python scripts (Extract_Ner.py, Extract_Ner_API.py) show that the project has an exploratory/training phase and an inference/API layer.
• The presence of API scripts (Extract_Ner_API.py) and a hello-world example (helloworld.py) suggest that once trained, models are integrated into an API service to perform NER and conversion tasks.
System Components
a. Data Ingestion & Preprocessing:
• Component: Data Sources (raw recipe text, taste sets in CSV/JSONL formats)
• Process: Clean, preprocess, and prepare the input data (as seen in notebooks like cleandata.ipynb and tasteset.csv/tasteset_final.jsonl).
b. Model Training & Experimentation:
• Component(s): Training Pipelines under “DevErrors” and “recipe_ner_model”
• Included items: Notebooks (train.ipynb, trainv2.ipynb, ver2bert.ipynb, etc.), fine-tuned models (DistilBERT variants), checkpoint directories and configuration files.
• Pattern: Iterate on model improvements using different experimental versions and checkpoints.
c. Named Entity Recognition (NER) Extraction Module:
• Component: NER Extraction Scripts and Notebooks (Extract_Ner.ipynb, Extract_Ner.py)
• Role: Process recipe texts to identify ingredients and measurements.
d. Inference & API Layer:
• Component: API Service (Extract_Ner_API.py) and supporting demo scripts (helloworld.py)
• Function: Provide real-time extraction and conversion for recipe data.
e. Optional/Supporting Components:
• Experiment Tracking & Logging: Folder “results” containing training logs and checkpoints.
• Additional utilities: Files like geminiapifetch.ipynb for data fetching, which may connect to external recipe sources or databases.
Establish the Relationships and Data Flows
• How outputs of training (model checkpoints, configuration files) feed into the Inference/NER module.
• NER Extraction module to the API layer, which then outputs processed ingredient data.
• Optionally, indicate the integration of a conversion algorithm (even if not explicitly coded, this is implied by the objective to “convert measurements into grams”) which uses NER output along with a (possibly external) database of ingredient densities.
Architectural Patterns and Technical Stack
• Modular separation between data preparation, model training, NER extraction, and API service.
• Use of transformer-based architectures (DistilBERT fine tuning) and beacon signs of PyTorch (e.g., rng_state.pth, training_args.bin) plus configuration files suggesting HuggingFace toolkit implementations.
• The iterative experimentation model (multiple versioned checkpoints and notebooks) suggests a research-first approach that later feeds into deployment.