How jmpsec/osctrl works
Purpose and executable surfaces
osctrl manages osquery fleets: configuration distribution, check-in/log ingestion, distributed queries, console/file operations, carves, alerts, posture, and fleet administration. Go entrypoints are TLS remote API (cmd/tls/main.go), operator REST API (cmd/api/main.go), CLI (cmd/cli/main.go), and standalone MCP stdio server (cmd/mcp/main.go). The browser SPA starts at frontend/src/main.tsx and routes via frontend/src/router.tsx.
Agent control and ingestion plane
osctrl-tls implements the osquery remote TLS API boundary. It handles agent enrollment/config/distributed-query reads and result/status log writes, node activity, carve transfers, and metrics. Request handlers are separated from shared domain logic in cmd/tls/handlers/handlers.go, cmd/tls/handlers/get.go, and cmd/tls/handlers/post.go. This service is the untrusted endpoint-facing boundary; agents do not directly access the operator API.
Shared fleet domain and persistence
Shared Go packages centralize business state used by API and TLS services. Environment configuration, package/config assembly, targeting, and cached environment state reside in pkg/environments/environments.go, pkg/environments/osqueryconf.go, and pkg/environments/env-cache.go. Node registration/check-ins and node cache are in pkg/nodes/nodes.go, pkg/nodes/checkins.go, and pkg/nodes/node-cache.go; query lifecycle/results/dispatch cache in pkg/queries/queries.go, pkg/queries/query-dispatch-cache.go, and pkg/queries/saved.go.
PostgreSQL is the default durable store through GORM, with MySQL/SQLite supported. Redis supplies typed caches, session hints, query dispatch state, settings, and activity tracking (pkg/cache/redis-json.go, pkg/activity/redis.go, pkg/settings/settings-cache.go).
Operator API, identity, and administration
osctrl-api exposes REST endpoints, API-facing auth, audit, configuration, and fleet administration. Routing/handler composition begins in cmd/api/handlers/handlers.go; fleet operations span cmd/api/handlers/nodes.go, cmd/api/handlers/queries.go, and cmd/api/handlers/environments.go. Default operator authentication is JWT, with OIDC/SAML provider support and MFA including TOTP, WebAuthn, and recovery codes (cmd/api/auth.go, pkg/authproviders/authproviders.go, pkg/mfa/manager.go). Permissions and environment scope are enforced at this boundary; audit events are managed by pkg/auditlog/audit.go.
UI, CLI, and MCP clients
The React/TypeScript SPA consumes the REST API through typed client modules such as frontend/src/api/client.ts, presenting environment-scoped fleet workflows under frontend/src/routes/_app/env/$env/route.tsx and feature pages such as frontend/src/features/nodes/NodeDetailPage.tsx. osctrl-cli is primarily an API client, while retaining direct-DB capability (cmd/cli/main.go, pkg/apiclient/client.go).
MCP tools are shared by hosted and standalone modes: pkg/mcp/server.go, pkg/mcp/tools_fleet.go, and pkg/mcp/tools_write.go. Standalone MCP authenticates to the API; hosted MCP is optional and write operations are separately gated.
Execution, file operations, and derived workflows
Distributed queries drive node console, accelerated file explorer, and carving: pkg/console/manager.go, pkg/fileexplorer/manager.go, and pkg/carves/carves.go. Carve storage supports database and S3 (pkg/carves/db.go, pkg/carves/s3.go). Optional posture derives scores from scheduled-query data (pkg/posture/posture.go, pkg/posture/scoring.go), and GeoIP enrichment is implemented in pkg/geoip/geoip.go.
Logs, alerts, health, and deployment
TLS log ingestion passes through pluggable logging/export paths (pkg/logging/process.go, pkg/logging/exporter_factory.go) supporting destinations including S3, Kafka, Kinesis, Elasticsearch, Splunk, and others. Alert matching/ingestion and asynchronous notification dispatch are in pkg/alerts/ingest.go, pkg/alerts/matcher.go, and pkg/alerts/worker.go; channels include webhook and email.
Health combines DB/Redis checks, service heartbeats, runtime metrics, and upgrade status (pkg/health/manager.go, pkg/health/runtime.go). Local deployment composes frontend, API, TLS, PostgreSQL, Redis, nginx, and sample agents in docker-compose-dev.yml; production configs and container definitions are under deploy/config/api.yml, deploy/config/tls.yml, and deploy/cicd/docker/.
Main entry points: queries.go Query lifecycle · handlers/handlers.go TLS handlers · main.go osctrl-api · handlers/handlers.go Fleet administration
Components 20
Clients
- osquery agents
- Browser SPA
frontend/src/main.tsx - osctrl CLI · Can also access the database directly.
cmd/cli/main.go - MCP server · Standalone mode authenticates to the operator API.
cmd/mcp/main.go
Service boundaries
- osctrl-tls · Untrusted agent-facing osquery boundary.
cmd/tls/main.go - TLS handlers · Separates remote API transport from shared domain logic.
cmd/tls/handlers/handlers.go - osctrl-api · Enforces permissions and environment scope.
cmd/api/main.go - Fleet administration
cmd/api/handlers/handlers.go - Identity and MFA · JWT default with OIDC, SAML, TOTP, WebAuthn, and recovery codes.
cmd/api/auth.go
Fleet domain
- Environment configuration · Builds packages/configuration, targeting, and cached environment state.
pkg/environments/environments.go - Node lifecycle · Registration, check-ins, and node cache.
pkg/nodes/nodes.go - Query lifecycle · Dispatches queries and retains results and saved queries.
pkg/queries/queries.go
Data and integrations
- Durable datastore · MySQL and SQLite are also supported.
- Redis caches · Stores dispatch, settings, sessions, and activity hints.
pkg/cache/redis-json.go - Log export pipeline · Exports to destinations such as S3, Kafka, Kinesis, Elasticsearch, and Splunk.
pkg/logging/process.go
Derived workflows
- Console and file operations · Console and accelerated file explorer use distributed queries.
pkg/console/manager.go - Carves · Stores carve data in the database or S3.
pkg/carves/carves.go - Alerts · Matches ingested events and dispatches webhook/email notifications asynchronously.
pkg/alerts/ingest.go - Posture scoring · Optionally derives scores from scheduled-query data.
pkg/posture/posture.go - Health and runtime · Combines datastore checks, heartbeats, metrics, and upgrade status.
pkg/health/manager.go
Connections 26
- osquery agents → osctrl-tls: TLS remote API
- osctrl-tls → TLS handlers: routes requests
- TLS handlers → Environment configuration: config and targeting
- TLS handlers → Node lifecycle: enrollment and check-ins
- TLS handlers → Query lifecycle: distributed reads and results
- TLS handlers → Log export pipeline: ingested logs
- TLS handlers → Carves: carve transfers
- Browser SPA → osctrl-api: REST
- osctrl CLI → osctrl-api: API client
- MCP server → osctrl-api: authenticated tools
- osctrl-api → Identity and MFA: auth and MFA
- osctrl-api → Fleet administration: routes fleet operations
- Fleet administration → Environment configuration: administers
- Fleet administration → Node lifecycle: administers
- Fleet administration → Query lifecycle: administers
- Environment configuration → Durable datastore: persists
- Node lifecycle → Durable datastore: persists
- Query lifecycle → Durable datastore: persists
- Environment configuration → Redis caches: cached state
- Node lifecycle → Redis caches: node/activity cache
- Query lifecycle → Redis caches: dispatch cache
- Query lifecycle → Console and file operations: executes operations
- Query lifecycle → Carves: drives collection
- Log export pipeline → Alerts: events
- Query lifecycle → Posture scoring: scheduled-query data
- osctrl-api → Health and runtime: reports service state