How idaholab/malcolm works
Project type and purpose
• Malcolm is a full-stack, container-based network traffic analysis platform.
• It ingests PCAPs, Zeek logs, Suricata alerts via web UI or live forwarders, normalizes/enriches them, stores them in an analytics store, and presents them through two key UIs: OpenSearch Dashboards and Arkime.
File structure
• Top-level “services” directories correspond to standalone containers:
– api (Django REST API)
– arkime (session capture & search UI)
– dashboards (OpenSearch Dashboards + helper scripts)
– file-upload, file-monitor, filebeat, pcap-capture, pcap-monitor, suricata, zeek, logstash, freq-server
– netbox (asset database & enrichment)
– keycloak (OIDC auth)
– htadmin (simple PHP/nginx admin UI)
– nginx (reverse-proxy & routing)
– opensearch (Elasticsearch-compatible analytics store)
– redis, postgresql (caching & relational storage)
• Orchestration files: docker-compose.yml and a full set of Kubernetes YAML under kubernetes/.
• Shared scripts and config under shared/, scripts/, config/, container-health-scripts/.
README for additional insights
• Emphasizes container-based deployment, secured communications, extensibility to ICS protocols, and two primary UIs.
• Mentions “cluster of software containers” with “simple scripts” for setup.
Detail without overthinking
• Distinct container as a component.
• Critical paths: PCAP ingestion, log forwarding, user requests.
• Integrations: NetBox enrichment, Keycloak auth, dashboards-helper template loading.