rudy-07/inferenceos

Purpose and Entry Points

InferenceOS is a local, hardware-aware LLM inference control plane layered over a vendored llama.cpp execution engine. It accepts GGUF models, adapts placement, memory, and scheduling to available hardware, and exposes CLI, Python, and HTTP interfaces. Primary launch surfaces are inferenceos/__main__.py, cli/main.py, and the standalone inferenceos.py.

Control Plane and Runtime Orchestration

The runtime lifecycle is coordinated by the adaptive engine: profile hardware, choose a backend and model variant, create placement/memory plans, initialize execution, then collect runtime statistics. The orchestration boundary separates Python policy/control logic from the native backend process and GGUF parsing.

• inference_runtime/runtime_engine.py

• inference_runtime/inference_session.py

• orchestrator/engine.py

Hardware Profiling and Backend Selection

Multi-vendor profiling models CPU, GPU, memory, storage, and interconnect resources. Its outputs guide backend selection across CUDA, ROCm/HIP, Metal, Vulkan, and CPU paths; iGPU-specific workload suitability and contention are separately modeled.

• profiler/hardware_profiler.py

• profiler/gpu_profiler.py

• inference_runtime/backend_selector.py

• igpu_support/suitability_evaluator.py

Placement, Memory, and Live Migration

Before loading/execution, layer placement optimizes CPU/dGPU/iGPU allocation using model descriptors and cost models. Memory budgeting and unified-memory accounting constrain the plan. During inference, monitoring and hysteresis-controlled migration can mutate layer placement under memory pressure. This is the core safety boundary for preventing resource exhaustion.

• layer_placement/placement_engine.py

• memory_budget/planner.py

• memory_manager/unified_memory_manager.py

• layer_migration/migration_coordinator.py

Request Execution, Scheduling, and KV State

Inference requests pass through context, memory, and microbatch schedulers; the asynchronous pipeline supplies task queues, worker pools, prefetching, and optional CUDA stream management. KV cache management owns allocation, quantization/compression, eviction policy, importance scoring, and telemetry, while runtime-memory preflight assesses allocation risk.

• scheduler/microbatch_scheduler.py

• async_scheduler/async_runtime_engine.py

• kv_manager/manager.py

• runtime_memory/preflight_guard.py

Native Backend Boundary

llama.cpp is the vendored C++ kernel/runtime dependency; Python-side wrappers construct arguments, manage subprocesses, parse/select GGUF models, and translate control-plane decisions into backend execution. This boundary is where model loading, tensor execution, prompt ingestion, and token generation occur.

• llama.cpp

• inference_runtime/process_manager.py

• orchestrator/backend_wrapper.py

• orchestrator/gguf_selector.py

Interfaces, Observability, and Feedback Loops

CLI commands support interactive runs, setup, profiling, optimization, and monitoring. The FastAPI server provides OpenAI-compatible and Ollama-compatible routes, request queuing, streaming SSE, and metrics. Execution telemetry feeds health monitoring, performance analysis, SQLite-backed runtime knowledge/learning, and optimizer recommendations for subsequent runs.

• server/api/app.py

• server/runtime_adapter/adapter.py

• runtime_learning/engine.py

• runtime_kb/database.py

• performance_intelligence/engine.py