# cisagov/thorium architecture

> Architecture diagram of the GitHub repository cisagov/thorium, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cisagov/thorium
- Repository: https://github.com/cisagov/thorium

## Overview

##### Project type and purpose
- This is a full-stack, microservices-oriented platform (“Thorium”) for orchestrating file analysis at scale.
- Front-end: React GUI (`ui/`).
- CLI tools: `thorctl` (user CLI), `thoradm` (admin CLI).
- API: Rust service (`api/`) exposing REST endpoints, backed by ScyllaDB, Redis, Elasticsearch, S3.
- Worker/microservices: `agent`, `event-handler`, `reactor`, `scaler`, `search-streamer`.
- Kubernetes operator: `operator/` crate plus `k8s/` manifests for on-cluster deployment.
- Local deployment helper: `minithor/`.

##### High-level components
a. Clients
- Web UI (React SPA)
- CLI tools (`thorctl`, `thoradm`)
b. Control plane (API & orchestration)
- REST API service
- Kubernetes Operator (manages cluster CRDs)
- Reactor (dispatch jobs to workers)
- Scaler (autoscale compute resources)
- Event-handler (listen for job-state changes, retries)
c. Data plane (work execution)
- Agent(s) (execute analysis in Docker/VM/shell)
d. Supporting infrastructure
- ScyllaDB (primary data store)
- Redis (caching, queues)
- Elasticsearch (full-text search index)
- S3 (blob storage for files/results)
- Kubernetes cluster (for pods, CRDs, ingress)

##### Relationships and data flows
1. UI/CLI → API (HTTP/HTTPS).
2. API writes job/task records to ScyllaDB, pushes to Redis queue.
3. Reactor subscribes to new-job events, enqueues to Event-handler or directly to Agent.
4. Event-handler orchestrates retries, state transitions.
5. Scaler monitors resource usage via K8s API, adjusts Agent pod counts.
6. Agent pulls tasks (from Redis/K8s queue), performs analysis, writes results to S3 & ScyllaDB.
7. Search-streamer reads new results, indexes into Elasticsearch.
8. API serves search queries to UI/CLI.
9. Operator watches CRDs, ensures cluster state (namespaces, deployments, secrets).

##### Architectural patterns
- Microservices communicating via HTTP and message queues.
- Event-driven job orchestration (Reactor + Event-handler).
- Autoscaling based on cluster metrics (Scaler).
- Kubernetes Operator pattern for CRD management.
- Hexagonal/Clean architecture in Rust crates (separate `libs`, `models`, `routes`).

##### Technologies & frameworks
- Rust + Actix/Web (API, microservices, operator).
- React + Vite (UI).
- ScyllaDB, Redis, Elasticsearch, S3.
- CLI using StructOpt/Clap (`thorctl`, `thoradm`).

## Diagram (Mermaid)

```mermaid
flowchart TD
    title["Thorium Architecture Diagram\nVersion 1.0\n2023-09-15"]

    subgraph "Clients" 
        direction TB
        UI["Web UI (React SPA)"]:::frontend
        Thorctl["User CLI (thorctl)"]:::frontend
        Thoradm["Admin CLI (thoradm)"]:::frontend
    end

    subgraph "Control Plane" 
        direction TB
        API["REST API Service"]:::control
        Reactor["Reactor"]:::control
        EventHandler["Event-handler"]:::control
        Scaler["Scaler"]:::control
        Operator["Kubernetes Operator"]:::control
    end

    subgraph "Data Plane" 
        direction TB
        Agent["Agent"]:::worker
        SearchStreamer["Search-streamer"]:::worker
    end

    subgraph "Platform Infrastructure" 
        direction TB
        ScyllaDB[(ScyllaDB)]:::infra
        Redis[(Redis)]:::infra
        ES[(Elasticsearch)]:::infra
        S3[(S3 Storage)]:::infra
        K8s[(Kubernetes Cluster)]:::infra
        K8sManifests["k8s/ (Manifests)"]:::infra
        Minithor["minithor/"]:::infra
    end

    %% Connections
    UI -->|HTTP/HTTPS| API
    Thorctl -->|HTTP/HTTPS| API
    Thoradm -->|HTTP/HTTPS| API

    API -->|Writes jobs| ScyllaDB
    API -->|Enqueues tasks| Redis

    Redis -.->|Pub/Sub| Reactor
    Reactor -->|Dispatches| EventHandler
    Reactor -->|Dispatches| Agent

    EventHandler -->|State updates| ScyllaDB
    EventHandler -->|Re-enqueue| Redis

    Scaler <-->|Monitors & scales| K8s

    Agent -->|Pull tasks| Redis
    Agent -->|Executes analysis| K8s
    Agent -->|Writes results| S3
    Agent -->|Writes metadata| ScyllaDB

    Agent -->|Emits results| SearchStreamer
    SearchStreamer -->|Indexes| ES

    API -->|Search queries| ES
    API -->|Manages CRDs| Operator
    Operator -->|Applies| K8s
    Minithor -->|Deploys Locally| K8s

    %% Legend
    subgraph "Legend"
        direction TB
        HTTP["HTTP/HTTPS"]:::legendArrow
        PUBSUB["Pub/Sub (Redis)"]:::legendArrow
        POLL["Poll/Bi-dir"]:::legendArrow
    end

    %% Click Events
    click UI "https://github.com/cisagov/thorium/tree/main/ui/"
    click Thorctl "https://github.com/cisagov/thorium/tree/main/thorctl/"
    click Thoradm "https://github.com/cisagov/thorium/tree/main/thoradm/"
    click API "https://github.com/cisagov/thorium/tree/main/api/"
    click Agent "https://github.com/cisagov/thorium/tree/main/agent/"
    click EventHandler "https://github.com/cisagov/thorium/tree/main/event-handler/"
    click Reactor "https://github.com/cisagov/thorium/tree/main/reactor/"
    click Scaler "https://github.com/cisagov/thorium/tree/main/scaler/"
    click SearchStreamer "https://github.com/cisagov/thorium/tree/main/search-streamer/"
    click Operator "https://github.com/cisagov/thorium/tree/main/operator/"
    click K8sManifests "https://github.com/cisagov/thorium/tree/main/k8s/"
    click Minithor "https://github.com/cisagov/thorium/tree/main/minithor/"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#0056b3,color:#0056b3;
    classDef control fill:#d4edda,stroke:#155724,color:#155724;
    classDef worker fill:#fff3cd,stroke:#856404,color:#856404;
    classDef infra fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef legendArrow stroke-dasharray: 2 2,stroke:#333,color:#333;
```
