# cactus-compute/needle architecture

> Architecture diagram of the GitHub repository cactus-compute/needle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cactus-compute/needle
- Repository: https://github.com/cactus-compute/needle

## Overview

#### Purpose and public entry points
Needle is a Python package for local, constrained tool-calling inference, structured extraction, LoRA fine-tuning, and export of compact `.cact` model binaries. Primary library APIs are exposed through `needle/__init__.py`; command-line commands are implemented by `needle/cli.py`. The optional browser playground is served by `needle/playground/server.py`.

#### Agent runtime and tool loop
`Needle` orchestrates a session: accept user text, select/render relevant tools, request constrained model completion, execute selected Python tools, feed results back, and return structured output plus confidence. Tool declaration/introspection and execution contracts are centered in `needle/agent/tools.py`, while engine acquisition/loading is separated into `needle/agent/fetch.py`.

#### Model execution boundary
The Python package does not implement the production inference engine itself; it fetches and caches a prebuilt engine/weights artifact, then invokes it for offline inference. This creates a hard boundary between Python-facing orchestration and the self-contained quantized binary runtime. Model architecture definitions and inference-side support live in `needle/model/architecture.py` and `needle/model/run.py`.

#### Constrained decoding and extraction
Tool arguments and extraction outputs are constrained to schemas through byte-level grammar-aware decoding, producing JSON-compatible structured results rather than unconstrained text. Tokenization and decoding are distinct stages in `needle/model/tokenizer.py` and `needle/model/decode.py`. The same runtime path supports direct completions, schema extraction, and agent tool calls.

#### Fine-tuning pipeline
Training uses JAX-based LoRA over a frozen base checkpoint. The pipeline consumes JSONL tool-call examples, optionally generates/augments examples via an external OpenAI-compatible endpoint, trains an adapter, then preserves deployment compatibility by merging it into the base before export. Core training logic is in `needle/model/finetune.py`; CLI-driven data synthesis is part of `needle/cli.py`.

#### Build, quantization, and distribution
Export merges a base checkpoint and optional LoRA adapter, quantizes model weights, and emits a portable `.cact` artifact consumable by the same inference engine. These transformations are separated into `needle/model/export.py` and `needle/model/quantize.py`. Artifacts may be downloaded from or uploaded to Hugging Face; engine/base checkpoint retrieval also depends on Hugging Face.

#### Playground service
The playground is a local server that initializes the model before accepting requests, maintains conversational follow-ups, and can invoke fine-tuning from the UI. Backend coordination is in `needle/playground/server.py`; the static browser client is `needle/playground/index.html` and `needle/playground/app.js`.

#### Runtimes and external services
Inference targets a small cached native/model engine and runs without network after setup. Fine-tuning requires Python/JAX, with optional CUDA or Apple Metal acceleration. Hugging Face supplies engine, base-model, and published `.cact` artifacts. Optional synthetic-data generation uses OpenRouter by default, or an environment-configured OpenAI-compatible gateway.

Main entry points: [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) (CLI commands), [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) (LoRA fine-tuning), [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) (Engine acquisition), [`needle/playground/server.py`](https://github.com/cactus-compute/needle/blob/main/needle/playground/server.py) (Playground server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public["Public Interfaces"]
  node_library["Needle library API<br/>Python package<br/>[__init__.py]"]
  node_cli["CLI commands<br/>Python CLI<br/>[cli.py]"]
end

subgraph group_runtime["Inference Runtime"]
  node_agent["Agent orchestrator<br/>session runtime"]
  node_tools["Tool contracts<br/>tool registry<br/>[tools.py]"]
  node_fetch["Engine acquisition<br/>artifact loader<br/>[fetch.py]"]
  node_native_engine{{"Cached native engine<br/>quantized binary runtime"}}
  node_architecture["Model architecture<br/>model definitions<br/>[architecture.py]"]
  node_run["Inference support<br/>model runtime support<br/>[run.py]"]
  node_tokenizer["Tokenization<br/>token encoder<br/>[tokenizer.py]"]
  node_decode["Grammar-constrained decoding<br/>structured decoder<br/>[decode.py]"]
end

subgraph group_training["Training &amp; Export"]
  node_finetune["LoRA fine-tuning<br/>JAX training<br/>[finetune.py]"]
  node_export["Model export<br/>artifact builder<br/>[export.py]"]
  node_quantize["Weight quantization<br/>model transform<br/>[quantize.py]"]
  node_cact[".cact artifact<br/>portable model binary"]
end

subgraph group_playground["Local Playground"]
  node_playground_server["Playground server<br/>local web server<br/>[server.py]"]
  node_playground_client["Browser playground<br/>static client<br/>[index.html]"]
  node_playground_app["Playground UI logic<br/>browser JavaScript<br/>[app.js]"]
end

node_huggingface[("Hugging Face<br/>artifact service")]
node_openai_gateway{{"OpenAI-compatible gateway<br/>synthetic-data service"}}

node_library -->|"creates sessions"| node_agent
node_cli -->|"runs inference"| node_agent
node_cli -->|"starts training"| node_finetune
node_cli -->|"exports models"| node_export
node_cli -.->|"optional data synthesis"| node_openai_gateway
node_agent -->|"selects and executes"| node_tools
node_agent -->|"acquires engine"| node_fetch
node_agent -->|"encodes requests"| node_tokenizer
node_agent -->|"requests structured output"| node_decode
node_fetch -->|"downloads artifacts"| node_huggingface
node_fetch -->|"caches and loads"| node_native_engine
node_architecture -->|"defines model"| node_run
node_tokenizer -->|"token input"| node_native_engine
node_native_engine -->|"token output"| node_decode
node_finetune -->|"retrieves base checkpoint"| node_huggingface
node_finetune -->|"produces LoRA adapter"| node_export
node_export -->|"quantizes merged weights"| node_quantize
node_quantize -->|"emits"| node_cact
node_cact -->|"loaded for inference"| node_native_engine
node_cact -.->|"optional publish"| node_huggingface
node_playground_client -->|"loads"| node_playground_app
node_playground_app -->|"local requests"| node_playground_server
node_playground_server -->|"serves conversations"| node_agent
node_playground_server -->|"starts UI training"| node_finetune

click node_library "https://github.com/cactus-compute/needle/blob/main/needle/__init__.py"
click node_cli "https://github.com/cactus-compute/needle/blob/main/needle/cli.py"
click node_tools "https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py"
click node_fetch "https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py"
click node_architecture "https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py"
click node_run "https://github.com/cactus-compute/needle/blob/main/needle/model/run.py"
click node_tokenizer "https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py"
click node_decode "https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py"
click node_finetune "https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py"
click node_export "https://github.com/cactus-compute/needle/blob/main/needle/model/export.py"
click node_quantize "https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py"
click node_playground_server "https://github.com/cactus-compute/needle/blob/main/needle/playground/server.py"
click node_playground_client "https://github.com/cactus-compute/needle/blob/main/needle/playground/index.html"
click node_playground_app "https://github.com/cactus-compute/needle/blob/main/needle/playground/app.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_library,node_cli toneBlue
class node_agent,node_tools,node_fetch,node_native_engine,node_architecture,node_run,node_tokenizer,node_decode toneAmber
class node_finetune,node_export,node_quantize,node_cact toneMint
class node_playground_server,node_playground_client,node_playground_app toneRose
class node_huggingface,node_openai_gateway toneNeutral
```

## Components

### Public Interfaces

- [Needle library API](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py): Python package · `needle/__init__.py`
- [CLI commands](https://github.com/cactus-compute/needle/blob/main/needle/cli.py): Python CLI · Runs inference, synthesis, fine-tuning, and export workflows. · `needle/cli.py`

### Inference Runtime

- **Agent orchestrator**: session runtime · Runs the constrained completion and tool-result feedback loop.
- [Tool contracts](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py): tool registry · Declares, introspects, renders, and executes Python tools. · `needle/agent/tools.py`
- [Engine acquisition](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py): artifact loader · Fetches and caches the native engine and model artifacts. · `needle/agent/fetch.py`
- **Cached native engine**: quantized binary runtime · Self-contained offline inference boundary.
- [Model architecture](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py): model definitions · `needle/model/architecture.py`
- [Inference support](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py): model runtime support · `needle/model/run.py`
- [Tokenization](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py): token encoder · `needle/model/tokenizer.py`
- [Grammar-constrained decoding](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py): structured decoder · Constrains tool arguments and extraction output to schemas. · `needle/model/decode.py`

### Training & Export

- [LoRA fine-tuning](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py): JAX training · Trains adapters over a frozen base checkpoint. · `needle/model/finetune.py`
- [Model export](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py): artifact builder · Merges the base model and optional adapter into a deployable artifact. · `needle/model/export.py`
- [Weight quantization](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py): model transform · Produces compact weights for portable artifacts. · `needle/model/quantize.py`
- **.cact artifact**: portable model binary · Consumed by the same native inference engine.

### Local Playground

- [Playground server](https://github.com/cactus-compute/needle/blob/main/needle/playground/server.py): local web server · Preloads the model, maintains follow-ups, and starts UI fine-tuning. · `needle/playground/server.py`
- [Browser playground](https://github.com/cactus-compute/needle/blob/main/needle/playground/index.html): static client · `needle/playground/index.html`
- [Playground UI logic](https://github.com/cactus-compute/needle/blob/main/needle/playground/app.js): browser JavaScript · `needle/playground/app.js`

### Other components

- **Hugging Face**: artifact service · Hosts engine, base checkpoints, and published artifacts.
- **OpenAI-compatible gateway**: synthetic-data service · OpenRouter is the default optional provider.

## Connections

- Needle library API → Agent orchestrator: creates sessions
- CLI commands → Agent orchestrator: runs inference
- CLI commands → LoRA fine-tuning: starts training
- CLI commands → Model export: exports models
- CLI commands → OpenAI-compatible gateway: optional data synthesis
- Agent orchestrator → Tool contracts: selects and executes
- Agent orchestrator → Engine acquisition: acquires engine
- Agent orchestrator → Tokenization: encodes requests
- Agent orchestrator → Grammar-constrained decoding: requests structured output
- Engine acquisition → Hugging Face: downloads artifacts
- Engine acquisition → Cached native engine: caches and loads
- Model architecture → Inference support: defines model
- Tokenization → Cached native engine: token input
- Cached native engine → Grammar-constrained decoding: token output
- LoRA fine-tuning → Hugging Face: retrieves base checkpoint
- LoRA fine-tuning → Model export: produces LoRA adapter
- Model export → Weight quantization: quantizes merged weights
- Weight quantization → .cact artifact: emits
- .cact artifact → Cached native engine: loaded for inference
- .cact artifact → Hugging Face: optional publish
- Browser playground → Playground UI logic: loads
- Playground UI logic → Playground server: local requests
- Playground server → Agent orchestrator: serves conversations
- Playground server → LoRA fine-tuning: starts UI training