# jmpsec/osctrl architecture

> Architecture diagram of the GitHub repository jmpsec/osctrl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jmpsec/osctrl
- Repository: https://github.com/jmpsec/osctrl

## Overview

#### Purpose and executable surfaces
osctrl manages osquery fleets: configuration distribution, check-in/log ingestion, distributed queries, console/file operations, carves, alerts, posture, and fleet administration. Go entrypoints are TLS remote API (`cmd/tls/main.go`), operator REST API (`cmd/api/main.go`), CLI (`cmd/cli/main.go`), and standalone MCP stdio server (`cmd/mcp/main.go`). The browser SPA starts at `frontend/src/main.tsx` and routes via `frontend/src/router.tsx`.

#### Agent control and ingestion plane
`osctrl-tls` implements the osquery remote TLS API boundary. It handles agent enrollment/config/distributed-query reads and result/status log writes, node activity, carve transfers, and metrics. Request handlers are separated from shared domain logic in `cmd/tls/handlers/handlers.go`, `cmd/tls/handlers/get.go`, and `cmd/tls/handlers/post.go`. This service is the untrusted endpoint-facing boundary; agents do not directly access the operator API.

#### Shared fleet domain and persistence
Shared Go packages centralize business state used by API and TLS services. Environment configuration, package/config assembly, targeting, and cached environment state reside in `pkg/environments/environments.go`, `pkg/environments/osqueryconf.go`, and `pkg/environments/env-cache.go`. Node registration/check-ins and node cache are in `pkg/nodes/nodes.go`, `pkg/nodes/checkins.go`, and `pkg/nodes/node-cache.go`; query lifecycle/results/dispatch cache in `pkg/queries/queries.go`, `pkg/queries/query-dispatch-cache.go`, and `pkg/queries/saved.go`.

PostgreSQL is the default durable store through GORM, with MySQL/SQLite supported. Redis supplies typed caches, session hints, query dispatch state, settings, and activity tracking (`pkg/cache/redis-json.go`, `pkg/activity/redis.go`, `pkg/settings/settings-cache.go`).

#### Operator API, identity, and administration
`osctrl-api` exposes REST endpoints, API-facing auth, audit, configuration, and fleet administration. Routing/handler composition begins in `cmd/api/handlers/handlers.go`; fleet operations span `cmd/api/handlers/nodes.go`, `cmd/api/handlers/queries.go`, and `cmd/api/handlers/environments.go`. Default operator authentication is JWT, with OIDC/SAML provider support and MFA including TOTP, WebAuthn, and recovery codes (`cmd/api/auth.go`, `pkg/authproviders/authproviders.go`, `pkg/mfa/manager.go`). Permissions and environment scope are enforced at this boundary; audit events are managed by `pkg/auditlog/audit.go`.

#### UI, CLI, and MCP clients
The React/TypeScript SPA consumes the REST API through typed client modules such as `frontend/src/api/client.ts`, presenting environment-scoped fleet workflows under `frontend/src/routes/_app/env/$env/route.tsx` and feature pages such as `frontend/src/features/nodes/NodeDetailPage.tsx`. `osctrl-cli` is primarily an API client, while retaining direct-DB capability (`cmd/cli/main.go`, `pkg/apiclient/client.go`).

MCP tools are shared by hosted and standalone modes: `pkg/mcp/server.go`, `pkg/mcp/tools_fleet.go`, and `pkg/mcp/tools_write.go`. Standalone MCP authenticates to the API; hosted MCP is optional and write operations are separately gated.

#### Execution, file operations, and derived workflows
Distributed queries drive node console, accelerated file explorer, and carving: `pkg/console/manager.go`, `pkg/fileexplorer/manager.go`, and `pkg/carves/carves.go`. Carve storage supports database and S3 (`pkg/carves/db.go`, `pkg/carves/s3.go`). Optional posture derives scores from scheduled-query data (`pkg/posture/posture.go`, `pkg/posture/scoring.go`), and GeoIP enrichment is implemented in `pkg/geoip/geoip.go`.

#### Logs, alerts, health, and deployment
TLS log ingestion passes through pluggable logging/export paths (`pkg/logging/process.go`, `pkg/logging/exporter_factory.go`) supporting destinations including S3, Kafka, Kinesis, Elasticsearch, Splunk, and others. Alert matching/ingestion and asynchronous notification dispatch are in `pkg/alerts/ingest.go`, `pkg/alerts/matcher.go`, and `pkg/alerts/worker.go`; channels include webhook and email.

Health combines DB/Redis checks, service heartbeats, runtime metrics, and upgrade status (`pkg/health/manager.go`, `pkg/health/runtime.go`). Local deployment composes frontend, API, TLS, PostgreSQL, Redis, nginx, and sample agents in `docker-compose-dev.yml`; production configs and container definitions are under `deploy/config/api.yml`, `deploy/config/tls.yml`, and `deploy/cicd/docker/`.

Main entry points: [`pkg/queries/queries.go`](https://github.com/jmpsec/osctrl/blob/develop/pkg/queries/queries.go) (Query lifecycle), [`cmd/tls/handlers/handlers.go`](https://github.com/jmpsec/osctrl/blob/develop/cmd/tls/handlers/handlers.go) (TLS handlers), [`cmd/api/main.go`](https://github.com/jmpsec/osctrl/blob/develop/cmd/api/main.go) (osctrl-api), [`cmd/api/handlers/handlers.go`](https://github.com/jmpsec/osctrl/blob/develop/cmd/api/handlers/handlers.go) (Fleet administration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Clients"]
  node_agents(("osquery agents<br/>agent fleet"))
  node_spa["Browser SPA<br/>React client<br/>[main.tsx]"]
  node_cli["osctrl CLI<br/>Go API client<br/>[main.go]"]
  node_mcp["MCP server<br/>stdio tool server<br/>[main.go]"]
end

subgraph group_services["Service boundaries"]
  node_tls["osctrl-tls<br/>TLS remote API<br/>[main.go]"]
  node_tls_handlers["TLS handlers<br/>request handlers<br/>[handlers.go]"]
  node_api["osctrl-api<br/>operator REST API<br/>[main.go]"]
  node_api_handlers["Fleet administration<br/>API handlers<br/>[handlers.go]"]
  node_identity["Identity and MFA<br/>authentication<br/>[auth.go]"]
end

subgraph group_domain["Fleet domain"]
  node_environments["Environment configuration<br/>fleet configuration<br/>[environments.go]"]
  node_nodes["Node lifecycle<br/>fleet state<br/>[nodes.go]"]
  node_queries["Query lifecycle<br/>distributed queries<br/>[queries.go]"]
end

subgraph group_data["Data and integrations"]
  node_postgres[("Durable datastore<br/>PostgreSQL via GORM")]
  node_redis[("Redis caches<br/>[redis-json.go]")]
  node_logging["Log export pipeline<br/>pluggable exporters<br/>[process.go]"]
end

subgraph group_workflows["Derived workflows"]
  node_console_files["Console and file operations<br/>query-driven operations<br/>[manager.go]"]
  node_carves["Carves<br/>file transfer workflow<br/>[carves.go]"]
  node_alerts["Alerts<br/>matching and notifications<br/>[ingest.go]"]
  node_posture["Posture scoring<br/>derived scoring<br/>[posture.go]"]
  node_health["Health and runtime<br/>service health<br/>[manager.go]"]
end

node_agents -->|"TLS remote API"| node_tls
node_tls -->|"routes requests"| node_tls_handlers
node_tls_handlers -->|"config and targeting"| node_environments
node_tls_handlers -->|"enrollment and check-ins"| node_nodes
node_tls_handlers -->|"distributed reads and results"| node_queries
node_tls_handlers -->|"ingested logs"| node_logging
node_tls_handlers -->|"carve transfers"| node_carves
node_spa -->|"REST"| node_api
node_cli -->|"API client"| node_api
node_mcp -->|"authenticated tools"| node_api
node_api -->|"auth and MFA"| node_identity
node_api -->|"routes fleet operations"| node_api_handlers
node_api_handlers -->|"administers"| node_environments
node_api_handlers -->|"administers"| node_nodes
node_api_handlers -->|"administers"| node_queries
node_environments -->|"persists"| node_postgres
node_nodes -->|"persists"| node_postgres
node_queries -->|"persists"| node_postgres
node_environments -->|"cached state"| node_redis
node_nodes -->|"node/activity cache"| node_redis
node_queries -->|"dispatch cache"| node_redis
node_queries -->|"executes operations"| node_console_files
node_queries -->|"drives collection"| node_carves
node_logging -->|"events"| node_alerts
node_queries -.->|"scheduled-query data"| node_posture
node_api -->|"reports service state"| node_health

click node_spa "https://github.com/jmpsec/osctrl/blob/develop/frontend/src/main.tsx"
click node_cli "https://github.com/jmpsec/osctrl/blob/develop/cmd/cli/main.go"
click node_mcp "https://github.com/jmpsec/osctrl/blob/develop/cmd/mcp/main.go"
click node_tls "https://github.com/jmpsec/osctrl/blob/develop/cmd/tls/main.go"
click node_tls_handlers "https://github.com/jmpsec/osctrl/blob/develop/cmd/tls/handlers/handlers.go"
click node_api "https://github.com/jmpsec/osctrl/blob/develop/cmd/api/main.go"
click node_api_handlers "https://github.com/jmpsec/osctrl/blob/develop/cmd/api/handlers/handlers.go"
click node_identity "https://github.com/jmpsec/osctrl/blob/develop/cmd/api/auth.go"
click node_environments "https://github.com/jmpsec/osctrl/blob/develop/pkg/environments/environments.go"
click node_nodes "https://github.com/jmpsec/osctrl/blob/develop/pkg/nodes/nodes.go"
click node_queries "https://github.com/jmpsec/osctrl/blob/develop/pkg/queries/queries.go"
click node_redis "https://github.com/jmpsec/osctrl/blob/develop/pkg/cache/redis-json.go"
click node_logging "https://github.com/jmpsec/osctrl/blob/develop/pkg/logging/process.go"
click node_console_files "https://github.com/jmpsec/osctrl/blob/develop/pkg/console/manager.go"
click node_carves "https://github.com/jmpsec/osctrl/blob/develop/pkg/carves/carves.go"
click node_alerts "https://github.com/jmpsec/osctrl/blob/develop/pkg/alerts/ingest.go"
click node_posture "https://github.com/jmpsec/osctrl/blob/develop/pkg/posture/posture.go"
click node_health "https://github.com/jmpsec/osctrl/blob/develop/pkg/health/manager.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_agents,node_spa,node_cli,node_mcp toneBlue
class node_tls,node_tls_handlers,node_api,node_api_handlers,node_identity toneAmber
class node_environments,node_nodes,node_queries toneMint
class node_postgres,node_redis,node_logging toneRose
class node_console_files,node_carves,node_alerts,node_posture,node_health toneIndigo
```

## Components

### Clients

- **osquery agents**: agent fleet
- [Browser SPA](https://github.com/jmpsec/osctrl/blob/develop/frontend/src/main.tsx): React client · `frontend/src/main.tsx`
- [osctrl CLI](https://github.com/jmpsec/osctrl/blob/develop/cmd/cli/main.go): Go API client · Can also access the database directly. · `cmd/cli/main.go`
- [MCP server](https://github.com/jmpsec/osctrl/blob/develop/cmd/mcp/main.go): stdio tool server · Standalone mode authenticates to the operator API. · `cmd/mcp/main.go`

### Service boundaries

- [osctrl-tls](https://github.com/jmpsec/osctrl/blob/develop/cmd/tls/main.go): TLS remote API · Untrusted agent-facing osquery boundary. · `cmd/tls/main.go`
- [TLS handlers](https://github.com/jmpsec/osctrl/blob/develop/cmd/tls/handlers/handlers.go): request handlers · Separates remote API transport from shared domain logic. · `cmd/tls/handlers/handlers.go`
- [osctrl-api](https://github.com/jmpsec/osctrl/blob/develop/cmd/api/main.go): operator REST API · Enforces permissions and environment scope. · `cmd/api/main.go`
- [Fleet administration](https://github.com/jmpsec/osctrl/blob/develop/cmd/api/handlers/handlers.go): API handlers · `cmd/api/handlers/handlers.go`
- [Identity and MFA](https://github.com/jmpsec/osctrl/blob/develop/cmd/api/auth.go): authentication · JWT default with OIDC, SAML, TOTP, WebAuthn, and recovery codes. · `cmd/api/auth.go`

### Fleet domain

- [Environment configuration](https://github.com/jmpsec/osctrl/blob/develop/pkg/environments/environments.go): fleet configuration · Builds packages/configuration, targeting, and cached environment state. · `pkg/environments/environments.go`
- [Node lifecycle](https://github.com/jmpsec/osctrl/blob/develop/pkg/nodes/nodes.go): fleet state · Registration, check-ins, and node cache. · `pkg/nodes/nodes.go`
- [Query lifecycle](https://github.com/jmpsec/osctrl/blob/develop/pkg/queries/queries.go): distributed queries · Dispatches queries and retains results and saved queries. · `pkg/queries/queries.go`

### Data and integrations

- **Durable datastore**: PostgreSQL via GORM · MySQL and SQLite are also supported.
- [Redis caches](https://github.com/jmpsec/osctrl/blob/develop/pkg/cache/redis-json.go): typed cache and activity state · Stores dispatch, settings, sessions, and activity hints. · `pkg/cache/redis-json.go`
- [Log export pipeline](https://github.com/jmpsec/osctrl/blob/develop/pkg/logging/process.go): pluggable exporters · Exports to destinations such as S3, Kafka, Kinesis, Elasticsearch, and Splunk. · `pkg/logging/process.go`

### Derived workflows

- [Console and file operations](https://github.com/jmpsec/osctrl/blob/develop/pkg/console/manager.go): query-driven operations · Console and accelerated file explorer use distributed queries. · `pkg/console/manager.go`
- [Carves](https://github.com/jmpsec/osctrl/blob/develop/pkg/carves/carves.go): file transfer workflow · Stores carve data in the database or S3. · `pkg/carves/carves.go`
- [Alerts](https://github.com/jmpsec/osctrl/blob/develop/pkg/alerts/ingest.go): matching and notifications · Matches ingested events and dispatches webhook/email notifications asynchronously. · `pkg/alerts/ingest.go`
- [Posture scoring](https://github.com/jmpsec/osctrl/blob/develop/pkg/posture/posture.go): derived scoring · Optionally derives scores from scheduled-query data. · `pkg/posture/posture.go`
- [Health and runtime](https://github.com/jmpsec/osctrl/blob/develop/pkg/health/manager.go): service health · Combines datastore checks, heartbeats, metrics, and upgrade status. · `pkg/health/manager.go`

## Connections

- osquery agents → osctrl-tls: TLS remote API
- osctrl-tls → TLS handlers: routes requests
- TLS handlers → Environment configuration: config and targeting
- TLS handlers → Node lifecycle: enrollment and check-ins
- TLS handlers → Query lifecycle: distributed reads and results
- TLS handlers → Log export pipeline: ingested logs
- TLS handlers → Carves: carve transfers
- Browser SPA → osctrl-api: REST
- osctrl CLI → osctrl-api: API client
- MCP server → osctrl-api: authenticated tools
- osctrl-api → Identity and MFA: auth and MFA
- osctrl-api → Fleet administration: routes fleet operations
- Fleet administration → Environment configuration: administers
- Fleet administration → Node lifecycle: administers
- Fleet administration → Query lifecycle: administers
- Environment configuration → Durable datastore: persists
- Node lifecycle → Durable datastore: persists
- Query lifecycle → Durable datastore: persists
- Environment configuration → Redis caches: cached state
- Node lifecycle → Redis caches: node/activity cache
- Query lifecycle → Redis caches: dispatch cache
- Query lifecycle → Console and file operations: executes operations
- Query lifecycle → Carves: drives collection
- Log export pipeline → Alerts: events
- Query lifecycle → Posture scoring: scheduled-query data
- osctrl-api → Health and runtime: reports service state
