abhaychandra123/toolcalllm

1. What this repo is

ToolCallLM is an end-to-end tool-calling fine-tuning pipeline centered on a base LLM (default Qwen3-8B). The core flow is: raw tool-call datasets in, standardized training examples out, LoRA/QLoRA fine-tuning, then inference, BFCL evaluation, and optional serving.

2. Configuration is the main control plane

The project is driven by YAML recipes in configs/. configs/base.yaml defines shared defaults, and concrete configs extend it for training, inference, evaluation, FP8/Unsloth variants, BFCL subsets, and experiment-specific runs. .env supplies path/model overrides such as DATA_DIR, OUTPUT_DIR, and BASE_MODEL.

3. Repository boundary: thin scripts, real logic in src/

scripts/ contains command-line entrypoints only. They load config, call one function in toolcall_lm.*, and exit. The actual architecture lives under src/toolcall_lm/, which is split into small subsystems: data, training, inference, api, eval, prompt, validators, tracking, and utils.

4. Data subsystem: normalize many dataset shapes into one schema

src/toolcall_lm/data/ converts multiple source formats into a unified tool-calling schema. It supports OpenAI-chat style, ShareGPT, Alpaca-like records, BFCL-specific conversions, and dataset-specific adapters for ToolACE and xLAM. The flow is raw JSONL/JSON in data/, normalization and filtering in preprocess.py, then processed JSONL/manifests under data/train/processed/.

5. Dataset quality gates happen early

The data pipeline includes schema validation, masking, truncation checks, and inspection utilities. Key boundaries are:

- schema.py / dataset.py for the canonical record shape

- preprocess.py for conversion and rejection logic

- masking.py, schema_jitter.py, thought_injection.py, and failure_driven_sampling.py for augmentation/experimentation

- inspect.py for human-readable validation before training

This repo treats bad training rows as a first-class failure mode, not a downstream surprise.

6. Prompting and output format are separate from the model

src/toolcall_lm/prompt/ defines how tool specs and chat templates are rendered. The repo explicitly supports Qwen-style <tool_call>...</tool_call> outputs and fallback JSON formatting through tool_format.py. This boundary matters because training, inference, and validation all depend on the exact tool-call wrapper.

7. Training subsystem: supervised fine-tuning with adapter-based methods

src/toolcall_lm/training/ wires the model, tokenizer, LoRA/QLoRA config, callbacks, and watchdogs into the training loop. The architecture is adapter-first: base model remains fixed, while checkpoints store LoRA adapters plus tokenizer/config artifacts. There are distinct recipes for BF16 LoRA, QLoRA, DPO, FP8, Unsloth, and recovery-oriented experiments.

8. Inference subsystem: same model, different boundary

src/toolcall_lm/inference/ provides a CLI-facing engine that loads the base model plus an optional adapter and produces either raw text or parsed tool calls. It is also where validation of model outputs happens via validators/json_parser.py and validators/tool_call_validator.py. The inference layer is the boundary between free-form generation and structured tool-call semantics.

9. Serving is a thin FastAPI wrapper over inference

src/toolcall_lm/api/server.py exposes health and chat-completions-style endpoints, with request/response schemas in api/schemas.py. The service reuses the same inference engine and prompt/tool formatting logic as the CLI, so the serving path is not a separate implementation. scripts/serve.py launches this API, and the README shows streaming support as an optional client-side concern.

10. Evaluation is BFCL-centric and produces its own artifact tree

src/toolcall_lm/eval/ contains the BFCL runner, Taubench runner, failure analysis, taxonomy, and run-comparison utilities. BFCL evaluation shells out to the official BFCL CLI, then aggregates category-level outputs into outputs/bfcl_results/ and outputs/bfcl_metrics/ style folders, including summaries and failure breakdowns. This is the main benchmark boundary for judging tool-calling quality.

11. Synthetic and experimental dataset generation is a separate lane

The synthetic_data_gen/ directory and several scripts under scripts/ build synthetic multiturn, recovery, preference, and negative samples. These artifacts feed back into the same preprocessing pipeline, but they are clearly separated from model training and evaluation so the repo can support both curated and generated data sources.

12. Operational helpers and infrastructure support

src/toolcall_lm/utils/ contains configuration loading, logging, seeding, checkpoint utilities, environment sanitization, notifier hooks, and small runtime patches. The project also includes integrations for W&B tracking, optional Telegram/status notifications, Dockerfiles for different runtime setups, and accelerate/bitsandbytes/Unsloth paths where memory or performance constraints matter.

13. Artifacts and state are intentionally kept out of source control

data/ holds downloaded/raw/processed datasets, while outputs/ is the workspace for checkpoints, merged models, BFCL results, metrics, logs, and generated reports. The many saved merged_* and bfcl_* folders in outputs/ show that experiments are reproducible by config and artifact path rather than by hidden runtime state.

14. Quality strategy: tests and diagnostic scripts

tests/ validates the critical seams: preprocessing, masking, schema jitter, JSON parsing, LoRA config, toolace pipeline integrity, BFCL-related behavior, and end-to-end dry runs. The numerous scripts in scripts/ act as targeted diagnostic tools for failure analysis, dataset audits, performance sweeps, and benchmark reproduction.

15. How to diagram the system

A good architecture diagram should show one central pipeline with these stages: raw data sources → preprocessing/validation → processed training set → adapter fine-tuning → adapter checkpoint/merged model → inference/serving → BFCL/Taubench evaluation and analysis. Surround that pipeline with the config layer, prompt/validator boundary, and utility/ops layer that supports each stage.