How bhangun/gollek works
Gollek is an AI inference engine and serving layer. Its user-facing workflow is to obtain or select a model, submit a prompt through the CLI or SDK, run inference on a supported backend, and return generated output; the CLI also exposes an API server. Major documented capabilities include model management, multimodal and speech/vision runners, quantization, and observability. The source excerpts show parts of runtime execution, model registration, and prefill/decode optimization; connections not established by those excerpts are included only where the README documents them.
Main entry points: ExecutionEngine.java Execution Runtime · InferenceEngine.java Inference SPI · gollek-cli CLI · GollekChatService.java SDK Chat
Components 20
Clients and Serving
- CLI
ui/gollek-cli - SDK Chat
framework/sdk/gollek-sdk-core/src/main/java/tech/kayys/gollek/sdk/core/GollekChatService.java - API Server
ui/gollek-api - gRPC Service
ui/gollek-sdk-protobuf/src/main/java/tech/kayys/gollek/protobuf/GollekGrpcService.java
Model Management
- Model Registry
framework/repository/gollek-model-database/src/main/java/tech/kayys/gollek/model/registry/ModelRegistryService.java - Model Download
framework/repository/gollek-model-database/src/main/java/tech/kayys/gollek/model/domain/download/ModelDownloadManager.java - Hugging Face
runtime/repository/gollek-model-repo-hf/src/main/java/tech/kayys/gollek/model/repo/hf/HuggingFaceRepository.java
Inference Runtime
- Inference Request
framework/spi/gollek-spi-inference/src/main/java/tech/kayys/gollek/spi/inference/InferenceRequest.java - Inference SPI
framework/spi/gollek-spi-inference/src/main/java/tech/kayys/gollek/spi/inference/InferenceEngine.java - Execution Runtime
runtime/core/gollek-runtime-core/src/main/java/tech/kayys/gollek/runtime/ExecutionEngine.java - Prefill Decode
runtime/optimization/gollek-plugin-prefill-decode/src/main/java/tech/kayys/gollek/prefilldecode/PrefillDecodeDisaggService.java - Tensor and IR Core
framework/core/gollek-core/src/main/java/tech/kayys/gollek/ir/GValueId.java - Inference Result
framework/spi/gollek-spi-inference/src/main/java/tech/kayys/gollek/spi/inference/InferenceResponse.java
Execution Backends
- GGUF Runner
runtime/runner/gguf/gollek-runner-gguf - SafeTensor Runner
runtime/runner/safetensor/gollek-safetensor-engine - ONNX Runner
runtime/runner/onnx/gollek-runner-onnx/src/main/java/tech/kayys/gollek/provider/onnx/OnnxEngine.java - LiteRT Runner
runtime/runner/litert/gollek-runner-litert - LibTorch Runner
runtime/runner/torch/gollek-runner-libtorch/src/main/java/tech/kayys/gollek/inference/libtorch/LibTorchEngine.java
Operations
- Metrics and Audit
framework/observability/gollek-observability
Other components
- User
Connections 22
- User → CLI: invokes
README.md - User → SDK Chat: uses
README.md - User → API Server: starts
README.md - CLI → Model Download: pulls models
README.md - Model Download → Hugging Face: downloads from
README.md - CLI → Model Registry: lists models
README.md - SDK Chat → Inference Request: builds request
framework/sdk/gollek-sdk-core/src/main/java/tech/kayys/gollek/sdk/core/GollekChatService.java - SDK Chat → Inference SPI: submits inference
- gRPC Service → Inference SPI: serves inference
ui/gollek-sdk-protobuf/src/main/java/tech/kayys/gollek/protobuf/GollekGrpcService.java - API Server → Inference SPI: serves inference
- Inference Request → Inference SPI: specifies input
framework/spi/gollek-spi-inference/src/main/java/tech/kayys/gollek/spi/inference/InferenceEngine.java - Inference SPI → Execution Runtime: executes via
- Execution Runtime → Tensor and IR Core: uses operators
runtime/core/gollek-runtime-core/src/main/java/tech/kayys/gollek/runtime/ExecutionEngine.java - Execution Runtime → SafeTensor Runner: dispatches to
- Execution Runtime → GGUF Runner: dispatches to
- Execution Runtime → ONNX Runner: dispatches to
- Execution Runtime → LiteRT Runner: dispatches to
- Execution Runtime → LibTorch Runner: dispatches to
- Prefill Decode → Inference Request: reads request
runtime/optimization/gollek-plugin-prefill-decode/src/main/java/tech/kayys/gollek/prefilldecode/PrefillDecodeDisaggService.java - Prefill Decode → Inference Result: produces output
runtime/optimization/gollek-plugin-prefill-decode/src/main/java/tech/kayys/gollek/prefilldecode/PrefillDecodeDisaggService.java - Inference SPI → Inference Result: returns result
framework/spi/gollek-spi-inference/src/main/java/tech/kayys/gollek/spi/inference/InferenceEngine.java - Inference SPI → Metrics and Audit: reports metrics
framework/observability/gollek-observability/src/main/java/tech/kayys/gollek/observability/InferenceEngineHealthCheck.java