How cisagov/thorium works
Project type and purpose
• This is a full-stack, microservices-oriented platform (“Thorium”) for orchestrating file analysis at scale.
• Front-end: React GUI (ui/).
• CLI tools: thorctl (user CLI), thoradm (admin CLI).
• API: Rust service (api/) exposing REST endpoints, backed by ScyllaDB, Redis, Elasticsearch, S3.
• Worker/microservices: agent, event-handler, reactor, scaler, search-streamer.
• Kubernetes operator: operator/ crate plus k8s/ manifests for on-cluster deployment.
• Local deployment helper: minithor/.
High-level components
a. Clients
• Web UI (React SPA)
• CLI tools (thorctl, thoradm)
b. Control plane (API & orchestration)
• REST API service
• Kubernetes Operator (manages cluster CRDs)
• Reactor (dispatch jobs to workers)
• Scaler (autoscale compute resources)
• Event-handler (listen for job-state changes, retries)
c. Data plane (work execution)
• Agent(s) (execute analysis in Docker/VM/shell)
d. Supporting infrastructure
• ScyllaDB (primary data store)
• Redis (caching, queues)
• Elasticsearch (full-text search index)
• S3 (blob storage for files/results)
• Kubernetes cluster (for pods, CRDs, ingress)
Relationships and data flows
1. UI/CLI → API (HTTP/HTTPS).
2. API writes job/task records to ScyllaDB, pushes to Redis queue.
3. Reactor subscribes to new-job events, enqueues to Event-handler or directly to Agent.
4. Event-handler orchestrates retries, state transitions.
5. Scaler monitors resource usage via K8s API, adjusts Agent pod counts.
6. Agent pulls tasks (from Redis/K8s queue), performs analysis, writes results to S3 & ScyllaDB.
7. Search-streamer reads new results, indexes into Elasticsearch.
8. API serves search queries to UI/CLI.
9. Operator watches CRDs, ensures cluster state (namespaces, deployments, secrets).
Architectural patterns
• Microservices communicating via HTTP and message queues.
• Event-driven job orchestration (Reactor + Event-handler).
• Autoscaling based on cluster metrics (Scaler).
• Kubernetes Operator pattern for CRD management.
• Hexagonal/Clean architecture in Rust crates (separate libs, models, routes).
Technologies & frameworks
• Rust + Actix/Web (API, microservices, operator).
• React + Vite (UI).
• ScyllaDB, Redis, Elasticsearch, S3.
• CLI using StructOpt/Clap (thorctl, thoradm).