# rudy-07/inferenceos architecture

> Architecture diagram of the GitHub repository rudy-07/inferenceos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rudy-07/inferenceos
- Repository: https://github.com/rudy-07/inferenceos

## Overview

#### Purpose and Entry Points
InferenceOS is a local, hardware-aware LLM inference control plane layered over a vendored `llama.cpp` execution engine. It accepts GGUF models, adapts placement, memory, and scheduling to available hardware, and exposes CLI, Python, and HTTP interfaces. Primary launch surfaces are `inferenceos/__main__.py`, `cli/main.py`, and the standalone `inferenceos.py`.

#### Control Plane and Runtime Orchestration
The runtime lifecycle is coordinated by the adaptive engine: profile hardware, choose a backend and model variant, create placement/memory plans, initialize execution, then collect runtime statistics. The orchestration boundary separates Python policy/control logic from the native backend process and GGUF parsing.
- `inference_runtime/runtime_engine.py`
- `inference_runtime/inference_session.py`
- `orchestrator/engine.py`

#### Hardware Profiling and Backend Selection
Multi-vendor profiling models CPU, GPU, memory, storage, and interconnect resources. Its outputs guide backend selection across CUDA, ROCm/HIP, Metal, Vulkan, and CPU paths; iGPU-specific workload suitability and contention are separately modeled.
- `profiler/hardware_profiler.py`
- `profiler/gpu_profiler.py`
- `inference_runtime/backend_selector.py`
- `igpu_support/suitability_evaluator.py`

#### Placement, Memory, and Live Migration
Before loading/execution, layer placement optimizes CPU/dGPU/iGPU allocation using model descriptors and cost models. Memory budgeting and unified-memory accounting constrain the plan. During inference, monitoring and hysteresis-controlled migration can mutate layer placement under memory pressure. This is the core safety boundary for preventing resource exhaustion.
- `layer_placement/placement_engine.py`
- `memory_budget/planner.py`
- `memory_manager/unified_memory_manager.py`
- `layer_migration/migration_coordinator.py`

#### Request Execution, Scheduling, and KV State
Inference requests pass through context, memory, and microbatch schedulers; the asynchronous pipeline supplies task queues, worker pools, prefetching, and optional CUDA stream management. KV cache management owns allocation, quantization/compression, eviction policy, importance scoring, and telemetry, while runtime-memory preflight assesses allocation risk.
- `scheduler/microbatch_scheduler.py`
- `async_scheduler/async_runtime_engine.py`
- `kv_manager/manager.py`
- `runtime_memory/preflight_guard.py`

#### Native Backend Boundary
`llama.cpp` is the vendored C++ kernel/runtime dependency; Python-side wrappers construct arguments, manage subprocesses, parse/select GGUF models, and translate control-plane decisions into backend execution. This boundary is where model loading, tensor execution, prompt ingestion, and token generation occur.
- `llama.cpp`
- `inference_runtime/process_manager.py`
- `orchestrator/backend_wrapper.py`
- `orchestrator/gguf_selector.py`

#### Interfaces, Observability, and Feedback Loops
CLI commands support interactive runs, setup, profiling, optimization, and monitoring. The FastAPI server provides OpenAI-compatible and Ollama-compatible routes, request queuing, streaming SSE, and metrics. Execution telemetry feeds health monitoring, performance analysis, SQLite-backed runtime knowledge/learning, and optimizer recommendations for subsequent runs.
- `server/api/app.py`
- `server/runtime_adapter/adapter.py`
- `runtime_learning/engine.py`
- `runtime_kb/database.py`
- `performance_intelligence/engine.py`

Main entry points: [`inference_runtime/runtime_engine.py`](https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/runtime_engine.py) (Adaptive Engine), [`inference_runtime/inference_session.py`](https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/inference_session.py) (Inference Session), [`profiler/hardware_profiler.py`](https://github.com/rudy-07/inferenceos/blob/main/profiler/hardware_profiler.py) (Hardware Profiler), [`memory_budget/planner.py`](https://github.com/rudy-07/inferenceos/blob/main/memory_budget/planner.py) (Memory Planner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Interfaces"]
  node_cli["CLI<br/>command interface<br/>[main.py]"]
  node_api["HTTP API<br/>FastAPI server<br/>[app.py]"]
  node_runtime_adapter["Runtime Adapter<br/>API bridge<br/>[adapter.py]"]
end

subgraph group_control["Control Plane"]
  node_adaptive_engine{{"Adaptive Engine<br/>runtime orchestrator<br/>[runtime_engine.py]"}}
  node_inference_session["Inference Session<br/>session lifecycle"]
  node_orchestrator["Orchestrator<br/>control policy<br/>[engine.py]"]
end

subgraph group_resources["Hardware &amp; Resource Safety"]
  node_hardware_profiler["Hardware Profiler<br/>resource discovery"]
  node_backend_selector["Backend Selector<br/>backend policy"]
  node_placement["Placement Engine<br/>layer planner"]
  node_memory_planner["Memory Planner<br/>budget planner<br/>[planner.py]"]
  node_migration["Migration Coordinator<br/>live placement control"]
  node_preflight["Memory Preflight<br/>allocation guard<br/>[preflight_guard.py]"]
end

subgraph group_execution["Execution Pipeline"]
  node_async_runtime["Async Runtime<br/>request pipeline"]
  node_microbatch["Microbatch Scheduler"]
  node_kv_manager["KV Cache Manager<br/>state manager<br/>[manager.py]"]
  node_backend_wrapper["Backend Wrapper<br/>native bridge<br/>[backend_wrapper.py]"]
  node_process_manager["Process Manager<br/>subprocess control<br/>[process_manager.py]"]
  node_llama_cpp{{"llama.cpp<br/>vendored native runtime<br/>[llama.cpp]"}}
end

subgraph group_feedback["Learning &amp; Feedback"]
  node_runtime_learning["Runtime Learning<br/>telemetry learner<br/>[engine.py]"]
  node_runtime_kb[("Runtime Knowledge Base<br/>SQLite knowledge store<br/>[database.py]")]
  node_performance["Performance Intelligence<br/>optimizer recommendations<br/>[engine.py]"]
end

node_cli -->|"launches"| node_adaptive_engine
node_api -->|"requests"| node_runtime_adapter
node_runtime_adapter -->|"queues"| node_async_runtime
node_adaptive_engine -->|"profiles"| node_hardware_profiler
node_hardware_profiler -->|"hardware profile"| node_backend_selector
node_hardware_profiler -->|"resource profile"| node_placement
node_adaptive_engine -->|"coordinates"| node_orchestrator
node_orchestrator -->|"creates"| node_inference_session
node_backend_selector -->|"backend choice"| node_backend_wrapper
node_placement -->|"placement plan"| node_memory_planner
node_memory_planner -->|"memory budget"| node_preflight
node_memory_planner -->|"approved plan"| node_inference_session
node_async_runtime -->|"schedules"| node_microbatch
node_microbatch -->|"allocation check"| node_preflight
node_microbatch -->|"KV state"| node_kv_manager
node_inference_session -->|"execution plan"| node_backend_wrapper
node_backend_wrapper -->|"launches"| node_process_manager
node_process_manager -->|"subprocess execution"| node_llama_cpp
node_kv_manager -->|"pressure telemetry"| node_migration
node_migration -.->|"mutates placement"| node_placement
node_inference_session -->|"runtime telemetry"| node_runtime_learning
node_runtime_learning -->|"stores outcomes"| node_runtime_kb
node_runtime_kb -->|"historical data"| node_performance
node_performance -.->|"recommendations"| node_adaptive_engine

click node_cli "https://github.com/rudy-07/inferenceos/blob/main/cli/main.py"
click node_api "https://github.com/rudy-07/inferenceos/blob/main/server/api/app.py"
click node_runtime_adapter "https://github.com/rudy-07/inferenceos/blob/main/server/runtime_adapter/adapter.py"
click node_adaptive_engine "https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/runtime_engine.py"
click node_inference_session "https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/inference_session.py"
click node_orchestrator "https://github.com/rudy-07/inferenceos/blob/main/orchestrator/engine.py"
click node_hardware_profiler "https://github.com/rudy-07/inferenceos/blob/main/profiler/hardware_profiler.py"
click node_backend_selector "https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/backend_selector.py"
click node_placement "https://github.com/rudy-07/inferenceos/blob/main/layer_placement/placement_engine.py"
click node_memory_planner "https://github.com/rudy-07/inferenceos/blob/main/memory_budget/planner.py"
click node_migration "https://github.com/rudy-07/inferenceos/blob/main/layer_migration/migration_coordinator.py"
click node_preflight "https://github.com/rudy-07/inferenceos/blob/main/runtime_memory/preflight_guard.py"
click node_async_runtime "https://github.com/rudy-07/inferenceos/blob/main/async_scheduler/async_runtime_engine.py"
click node_microbatch "https://github.com/rudy-07/inferenceos/blob/main/scheduler/microbatch_scheduler.py"
click node_kv_manager "https://github.com/rudy-07/inferenceos/blob/main/kv_manager/manager.py"
click node_backend_wrapper "https://github.com/rudy-07/inferenceos/blob/main/orchestrator/backend_wrapper.py"
click node_process_manager "https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/process_manager.py"
click node_llama_cpp "https://github.com/rudy-07/inferenceos/blob/main/llama.cpp"
click node_runtime_learning "https://github.com/rudy-07/inferenceos/blob/main/runtime_learning/engine.py"
click node_runtime_kb "https://github.com/rudy-07/inferenceos/blob/main/runtime_kb/database.py"
click node_performance "https://github.com/rudy-07/inferenceos/blob/main/performance_intelligence/engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_api,node_runtime_adapter toneBlue
class node_adaptive_engine,node_inference_session,node_orchestrator toneAmber
class node_hardware_profiler,node_backend_selector,node_placement,node_memory_planner,node_migration,node_preflight toneMint
class node_async_runtime,node_microbatch,node_kv_manager,node_backend_wrapper,node_process_manager,node_llama_cpp toneRose
class node_runtime_learning,node_runtime_kb,node_performance toneIndigo
```

## Components

### Interfaces

- [CLI](https://github.com/rudy-07/inferenceos/blob/main/cli/main.py): command interface · Interactive, setup, profiling, optimization, and monitoring commands. · `cli/main.py`
- [HTTP API](https://github.com/rudy-07/inferenceos/blob/main/server/api/app.py): FastAPI server · OpenAI- and Ollama-compatible streaming request surface. · `server/api/app.py`
- [Runtime Adapter](https://github.com/rudy-07/inferenceos/blob/main/server/runtime_adapter/adapter.py): API bridge · `server/runtime_adapter/adapter.py`

### Control Plane

- [Adaptive Engine](https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/runtime_engine.py): runtime orchestrator · Coordinates profiling, planning, backend initialization, and runtime statistics. · `inference_runtime/runtime_engine.py`
- [Inference Session](https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/inference_session.py): session lifecycle · `inference_runtime/inference_session.py`
- [Orchestrator](https://github.com/rudy-07/inferenceos/blob/main/orchestrator/engine.py): control policy · `orchestrator/engine.py`

### Hardware & Resource Safety

- [Hardware Profiler](https://github.com/rudy-07/inferenceos/blob/main/profiler/hardware_profiler.py): resource discovery · Profiles CPU, GPU, memory, storage, and interconnects. · `profiler/hardware_profiler.py`
- [Backend Selector](https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/backend_selector.py): backend policy · Selects CUDA, ROCm/HIP, Metal, Vulkan, or CPU execution. · `inference_runtime/backend_selector.py`
- [Placement Engine](https://github.com/rudy-07/inferenceos/blob/main/layer_placement/placement_engine.py): layer planner · Optimizes layer allocation across CPU, dGPU, and iGPU. · `layer_placement/placement_engine.py`
- [Memory Planner](https://github.com/rudy-07/inferenceos/blob/main/memory_budget/planner.py): budget planner · Constrains plans with memory budgets and unified-memory accounting. · `memory_budget/planner.py`
- [Migration Coordinator](https://github.com/rudy-07/inferenceos/blob/main/layer_migration/migration_coordinator.py): live placement control · Moves layers under memory pressure using monitored hysteresis. · `layer_migration/migration_coordinator.py`
- [Memory Preflight](https://github.com/rudy-07/inferenceos/blob/main/runtime_memory/preflight_guard.py): allocation guard · Assesses runtime allocation risk before execution. · `runtime_memory/preflight_guard.py`

### Execution Pipeline

- [Async Runtime](https://github.com/rudy-07/inferenceos/blob/main/async_scheduler/async_runtime_engine.py): request pipeline · Provides task queues, worker pools, prefetching, and optional CUDA streams. · `async_scheduler/async_runtime_engine.py`
- [Microbatch Scheduler](https://github.com/rudy-07/inferenceos/blob/main/scheduler/microbatch_scheduler.py): batch scheduler · `scheduler/microbatch_scheduler.py`
- [KV Cache Manager](https://github.com/rudy-07/inferenceos/blob/main/kv_manager/manager.py): state manager · Owns KV allocation, compression, eviction, importance, and telemetry. · `kv_manager/manager.py`
- [Backend Wrapper](https://github.com/rudy-07/inferenceos/blob/main/orchestrator/backend_wrapper.py): native bridge · Translates control decisions into backend arguments and model execution. · `orchestrator/backend_wrapper.py`
- [Process Manager](https://github.com/rudy-07/inferenceos/blob/main/inference_runtime/process_manager.py): subprocess control · `inference_runtime/process_manager.py`
- [llama.cpp](https://github.com/rudy-07/inferenceos/blob/main/llama.cpp): vendored native runtime · Loads GGUF models and performs tensor execution and token generation. · `llama.cpp`

### Learning & Feedback

- [Runtime Learning](https://github.com/rudy-07/inferenceos/blob/main/runtime_learning/engine.py): telemetry learner · Uses execution outcomes to improve subsequent runs. · `runtime_learning/engine.py`
- [Runtime Knowledge Base](https://github.com/rudy-07/inferenceos/blob/main/runtime_kb/database.py): SQLite knowledge store · `runtime_kb/database.py`
- [Performance Intelligence](https://github.com/rudy-07/inferenceos/blob/main/performance_intelligence/engine.py): optimizer recommendations · `performance_intelligence/engine.py`

## Connections

- CLI → Adaptive Engine: launches
- HTTP API → Runtime Adapter: requests
- Runtime Adapter → Async Runtime: queues
- Adaptive Engine → Hardware Profiler: profiles
- Hardware Profiler → Backend Selector: hardware profile
- Hardware Profiler → Placement Engine: resource profile
- Adaptive Engine → Orchestrator: coordinates
- Orchestrator → Inference Session: creates
- Backend Selector → Backend Wrapper: backend choice
- Placement Engine → Memory Planner: placement plan
- Memory Planner → Memory Preflight: memory budget
- Memory Planner → Inference Session: approved plan
- Async Runtime → Microbatch Scheduler: schedules
- Microbatch Scheduler → Memory Preflight: allocation check
- Microbatch Scheduler → KV Cache Manager: KV state
- Inference Session → Backend Wrapper: execution plan
- Backend Wrapper → Process Manager: launches
- Process Manager → llama.cpp: subprocess execution
- KV Cache Manager → Migration Coordinator: pressure telemetry
- Migration Coordinator → Placement Engine: mutates placement
- Inference Session → Runtime Learning: runtime telemetry
- Runtime Learning → Runtime Knowledge Base: stores outcomes
- Runtime Knowledge Base → Performance Intelligence: historical data
- Performance Intelligence → Adaptive Engine: recommendations