cactus-compute/needle

Purpose and public entry points

Needle is a Python package for local, constrained tool-calling inference, structured extraction, LoRA fine-tuning, and export of compact .cact model binaries. Primary library APIs are exposed through needle/__init__.py; command-line commands are implemented by needle/cli.py. The optional browser playground is served by needle/playground/server.py.

Agent runtime and tool loop

Needle orchestrates a session: accept user text, select/render relevant tools, request constrained model completion, execute selected Python tools, feed results back, and return structured output plus confidence. Tool declaration/introspection and execution contracts are centered in needle/agent/tools.py, while engine acquisition/loading is separated into needle/agent/fetch.py.

Model execution boundary

The Python package does not implement the production inference engine itself; it fetches and caches a prebuilt engine/weights artifact, then invokes it for offline inference. This creates a hard boundary between Python-facing orchestration and the self-contained quantized binary runtime. Model architecture definitions and inference-side support live in needle/model/architecture.py and needle/model/run.py.

Constrained decoding and extraction

Tool arguments and extraction outputs are constrained to schemas through byte-level grammar-aware decoding, producing JSON-compatible structured results rather than unconstrained text. Tokenization and decoding are distinct stages in needle/model/tokenizer.py and needle/model/decode.py. The same runtime path supports direct completions, schema extraction, and agent tool calls.

Fine-tuning pipeline

Training uses JAX-based LoRA over a frozen base checkpoint. The pipeline consumes JSONL tool-call examples, optionally generates/augments examples via an external OpenAI-compatible endpoint, trains an adapter, then preserves deployment compatibility by merging it into the base before export. Core training logic is in needle/model/finetune.py; CLI-driven data synthesis is part of needle/cli.py.

Build, quantization, and distribution

Export merges a base checkpoint and optional LoRA adapter, quantizes model weights, and emits a portable .cact artifact consumable by the same inference engine. These transformations are separated into needle/model/export.py and needle/model/quantize.py. Artifacts may be downloaded from or uploaded to Hugging Face; engine/base checkpoint retrieval also depends on Hugging Face.

Playground service

The playground is a local server that initializes the model before accepting requests, maintains conversational follow-ups, and can invoke fine-tuning from the UI. Backend coordination is in needle/playground/server.py; the static browser client is needle/playground/index.html and needle/playground/app.js.

Runtimes and external services

Inference targets a small cached native/model engine and runs without network after setup. Fine-tuning requires Python/JAX, with optional CUDA or Apple Metal acceleration. Hugging Face supplies engine, base-model, and published .cact artifacts. Optional synthetic-data generation uses OpenRouter by default, or an environment-configured OpenAI-compatible gateway.