# idaholab/malcolm architecture

> Architecture diagram of the GitHub repository idaholab/malcolm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/idaholab/malcolm
- Repository: https://github.com/idaholab/malcolm

## Overview

##### Project type and purpose
- Malcolm is a full-stack, container-based network traffic analysis platform.
- It ingests PCAPs, Zeek logs, Suricata alerts via web UI or live forwarders, normalizes/enriches them, stores them in an analytics store, and presents them through two key UIs: OpenSearch Dashboards and Arkime.

##### File structure
- Top-level “services” directories correspond to standalone containers:
– api (Django REST API)
– arkime (session capture & search UI)
– dashboards (OpenSearch Dashboards + helper scripts)
– file-upload, file-monitor, filebeat, pcap-capture, pcap-monitor, suricata, zeek, logstash, freq-server
– netbox (asset database & enrichment)
– keycloak (OIDC auth)
– htadmin (simple PHP/nginx admin UI)
– nginx (reverse-proxy & routing)
– opensearch (Elasticsearch-compatible analytics store)
– redis, postgresql (caching & relational storage)
- Orchestration files: docker-compose.yml and a full set of Kubernetes YAML under kubernetes/.
- Shared scripts and config under shared/, scripts/, config/, container-health-scripts/.

##### README for additional insights
- Emphasizes container-based deployment, secured communications, extensibility to ICS protocols, and two primary UIs.
- Mentions “cluster of software containers” with “simple scripts” for setup.

##### Detail without overthinking
- Distinct container as a component.
- Critical paths: PCAP ingestion, log forwarding, user requests.
- Integrations: NetBox enrichment, Keycloak auth, dashboards-helper template loading.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External
    internet["Clients & Forwarders"]:::external
    internet -->|HTTP, PCAP Logs, SYSLOG| nginx

    subgraph "Edge Layer"
        nginx["NGINX Reverse Proxy"]:::container
        keycloak["Keycloak (Auth)"]:::auth
    end

    nginx -->|Auth Redirect| keycloak
    nginx -->|Route to API/UI| api
    nginx -->|Route to Dashboards UI| dashboards
    nginx -->|Route to Arkime UI| arkime_ui
    nginx -->|Route to HTAdmin| htadmin

    subgraph "Ingestion & Processing"
        file_upload["PCAP Upload Service"]:::container
        file_monitor["PCAP Directory Watcher"]:::container
        filebeat["Filebeat Log Forwarder"]:::container
        suricata["Suricata IDS"]:::container
        zeek["Zeek Packet Analysis"]:::container
        logstash["Logstash Pipelines"]:::container
        pcap_capture["Live PCAP Capture"]:::container
        pcap_monitor["Live PCAP Watcher"]:::container
        freq_server["Frequency Server"]:::container
    end

    nginx -->|Web Upload| file_upload
    nginx -->|Watch Directory| file_monitor
    internet -->|SYSLOG| filebeat
    file_upload -->|PCAP Stream| logstash
    file_monitor -->|New PCAP| logstash
    filebeat -->|Logs| logstash
    suricata -->|Alerts| logstash
    zeek -->|Logs| logstash
    logstash -->|Enriched Data| opensearch
    pcap_capture -->|Captured Sessions| arkime_store
    pcap_monitor -->|Live Sessions| arkime_store
    freq_server -->|Stats| opensearch

    subgraph "Data Stores"
        opensearch["OpenSearch Cluster"]:::db
        postgres["PostgreSQL Metadata Store"]:::db
        redis["Redis Cache & Queues"]:::db
        arkime_store["Arkime Data Store"]:::db
        netbox["NetBox Asset DB"]:::db
    end

    subgraph "Presentation Layer"
        dashboards["OpenSearch Dashboards"]:::container
        arkime_ui["Arkime UI"]:::container
        htadmin["HTAdmin"]:::container
        api["Malcolm API (Django REST)"]:::container
    end

    dashboards -->|Query Data| opensearch
    arkime_ui -->|Session Search| arkime_store
    htadmin -->|Management API| api
    api -->|Metadata| postgres
    api -->|Cache/Tasks| redis
    api -->|Enrichment| netbox
    logstash -->|Lookup| netbox

    subgraph "Orchestration & Deployment"
        compose["Docker Compose"]:::orc
        kube["Kubernetes Manifests"]:::orc
    end

    subgraph "Shared & Config"
        shared_scripts["Shared Scripts & Helpers"]:::shared
        config_templates["Global Config Templates"]:::shared
        health_scripts["Container Health-check Scripts"]:::shared
        scripts_cli["CLI & Orchestration Scripts"]:::shared
    end

    %% Click Events
    click nginx "https://github.com/idaholab/malcolm/tree/main/nginx/"
    click keycloak "https://github.com/idaholab/malcolm/tree/main/keycloak/"
    click dashboards "https://github.com/idaholab/malcolm/tree/main/dashboards/"
    click arkime_ui "https://github.com/idaholab/malcolm/tree/main/arkime/"
    click htadmin "https://github.com/idaholab/malcolm/tree/main/htadmin/"
    click api "https://github.com/idaholab/malcolm/tree/main/api/"
    click file_upload "https://github.com/idaholab/malcolm/tree/main/file-upload/"
    click file_monitor "https://github.com/idaholab/malcolm/tree/main/file-monitor/"
    click pcap_capture "https://github.com/idaholab/malcolm/tree/main/pcap-capture/"
    click pcap_monitor "https://github.com/idaholab/malcolm/tree/main/pcap-monitor/"
    click filebeat "https://github.com/idaholab/malcolm/tree/main/filebeat/"
    click suricata "https://github.com/idaholab/malcolm/tree/main/suricata/"
    click zeek "https://github.com/idaholab/malcolm/tree/main/zeek/"
    click logstash "https://github.com/idaholab/malcolm/tree/main/logstash/"
    click freq_server "https://github.com/idaholab/malcolm/tree/main/freq-server/"
    click opensearch "https://github.com/idaholab/malcolm/tree/main/opensearch/"
    click postgres "https://github.com/idaholab/malcolm/tree/main/postgres-scripts/"
    click redis "https://github.com/idaholab/malcolm/blob/main/Dockerfiles/redis.Dockerfile"
    click arkime_store "https://github.com/idaholab/malcolm/tree/main/arkime/etc/"
    click netbox "https://github.com/idaholab/malcolm/tree/main/netbox/"
    click compose "https://github.com/idaholab/malcolm/blob/main/docker-compose.yml"
    click kube "https://github.com/idaholab/malcolm/tree/main/kubernetes/"
    click shared_scripts "https://github.com/idaholab/malcolm/tree/main/shared/"
    click config_templates "https://github.com/idaholab/malcolm/tree/main/config/"
    click health_scripts "https://github.com/idaholab/malcolm/tree/main/container-health-scripts/"
    click scripts_cli "https://github.com/idaholab/malcolm/tree/main/scripts/"

    %% Styles
    classDef container fill:#D6EFFF,stroke:#0366d6
    classDef db fill:#E0F2F1,stroke:#00796B
    classDef external fill:#ECEFF1,stroke:#607D8B
    classDef auth fill:#FFF3E0,stroke:#FB8C00
    classDef orc fill:#EDE7F6,stroke:#5E35B1
    classDef shared fill:#FFF9C4,stroke:#FBC02D
```
