# cisagov/malcolm architecture

> Architecture diagram of the GitHub repository cisagov/malcolm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cisagov/malcolm
- Repository: https://github.com/cisagov/malcolm

## Overview

Malcolm is a deployable network-security monitoring suite: users submit PCAPs, Zeek logs, or Suricata alerts—or forward/capture traffic—and analyze normalized, enriched results in OpenSearch Dashboards and Arkime. The graph preserves the documented ingestion, processing, enrichment, storage, and analysis stages, plus file analysis and deployment configuration. README descriptions support the broad workflow; sampled source confirms upload handling, queued PCAP processing, API access, and kiosk service control, but does not establish every container-to-container connection.

Main entry points: [`shared/bin/pcap_processor.py`](https://github.com/cisagov/malcolm/blob/main/shared/bin/pcap_processor.py) (PCAP Processor), [`dashboards`](https://github.com/cisagov/malcolm/tree/main/dashboards) (OpenSearch), [`logstash`](https://github.com/cisagov/malcolm/tree/main/logstash) (Normalization Enrichment), [`zeek`](https://github.com/cisagov/malcolm/tree/main/zeek) (Zeek Analysis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Traffic Ingestion"]
  node_upload["File Upload<br/>[submit.php]"]
  node_watchers["Upload Watchers"]
  node_pcapwatch["PCAP Watcher<br/>[pcap_watcher.py]"]
end

subgraph group_processing["Processing Enrichment"]
  node_pcapproc["PCAP Processor<br/>[pcap_processor.py]"]
  node_zeek["Zeek Analysis"]
  node_suricata["Suricata Analysis"]
  node_filescan["File Scanning"]
  node_logstash["Normalization Enrichment"]
  node_netbox[("Asset Inventory")]
end

subgraph group_storage["Data Services"]
  node_opensearch[("OpenSearch")]
end

subgraph group_analysis["Analysis Interfaces"]
  node_arkime["Arkime Sessions"]
  node_wise["WISE Fields<br/>[source.zeeklogs.js]"]
  node_dashboards["Dashboards"]
  node_api["Malcolm API<br/>[__init__.py]"]
end

subgraph group_operations["Deployment Operations"]
  node_installer["Installer Configuration"]
  node_kiosk["Kiosk Controls<br/>[routes.py]"]
  node_service["Service Operations<br/>[sys_service.py]"]
end

node_analyst(("Analyst"))
node_forwarder(("Remote Forwarder"))
node_network(("Network Traffic"))
node_browser(("Browser User"))

node_browser -->|"uploads artifacts"| node_upload
node_upload -->|"saves uploads"| node_watchers
node_forwarder -->|"forwards logs"| node_zeek
node_network -->|"captures traffic"| node_zeek
node_network -->|"captures traffic"| node_suricata
node_pcapwatch -->|"queues captures"| node_pcapproc
node_pcapproc -->|"dispatches analysis"| node_zeek
node_pcapproc -->|"dispatches analysis"| node_suricata
node_zeek -->|"emits logs"| node_logstash
node_suricata -->|"emits alerts"| node_logstash
node_logstash -.->|"looks up assets"| node_netbox
node_logstash -->|"writes events"| node_opensearch
node_pcapproc -->|"stores session data"| node_opensearch
node_pcapproc -->|"scans extracted files"| node_filescan
node_opensearch -->|"serves searches"| node_api
node_api -->|"provides data"| node_dashboards
node_opensearch -->|"serves visualizations"| node_dashboards
node_opensearch -->|"serves session data"| node_arkime
node_wise -->|"registers fields"| node_arkime
node_analyst -->|"explores traffic"| node_dashboards
node_analyst -->|"searches sessions"| node_arkime
node_browser -->|"requests controls"| node_kiosk
node_kiosk -->|"dispatches command"| node_service

click node_upload "https://github.com/cisagov/malcolm/blob/main/file-upload/php/submit.php"
click node_watchers "https://github.com/cisagov/malcolm/blob/main/filebeat/scripts/filebeat-watch-zeeklogs-uploads-folder.py"
click node_pcapwatch "https://github.com/cisagov/malcolm/blob/main/shared/bin/pcap_watcher.py"
click node_pcapproc "https://github.com/cisagov/malcolm/blob/main/shared/bin/pcap_processor.py"
click node_zeek "https://github.com/cisagov/malcolm/tree/main/zeek"
click node_suricata "https://github.com/cisagov/malcolm/tree/main/suricata"
click node_filescan "https://github.com/cisagov/malcolm/tree/main/filescan/python-filescan/filescan"
click node_logstash "https://github.com/cisagov/malcolm/tree/main/logstash"
click node_netbox "https://github.com/cisagov/malcolm/tree/main/netbox"
click node_opensearch "https://github.com/cisagov/malcolm/tree/main/dashboards"
click node_arkime "https://github.com/cisagov/malcolm/tree/main/arkime"
click node_wise "https://github.com/cisagov/malcolm/blob/main/arkime/wise/source.zeeklogs.js"
click node_dashboards "https://github.com/cisagov/malcolm/tree/main/dashboards"
click node_api "https://github.com/cisagov/malcolm/blob/main/api/project/__init__.py"
click node_installer "https://github.com/cisagov/malcolm/tree/main/scripts/installer"
click node_kiosk "https://github.com/cisagov/malcolm/blob/main/malcolm-iso/kiosk/interface/routes.py"
click node_service "https://github.com/cisagov/malcolm/blob/main/malcolm-iso/kiosk/interface/sysquery/sys_service.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_upload,node_watchers,node_pcapwatch,node_browser toneBlue
class node_pcapproc,node_zeek,node_suricata,node_filescan,node_logstash,node_netbox toneAmber
class node_opensearch toneMint
class node_arkime,node_wise,node_dashboards,node_api toneRose
class node_installer,node_kiosk,node_service,node_analyst,node_forwarder,node_network toneIndigo
```

## Components

### Traffic Ingestion

- [File Upload](https://github.com/cisagov/malcolm/blob/main/file-upload/php/submit.php): `file-upload/php/submit.php`
- [Upload Watchers](https://github.com/cisagov/malcolm/blob/main/filebeat/scripts/filebeat-watch-zeeklogs-uploads-folder.py): `filebeat/scripts/filebeat-watch-zeeklogs-uploads-folder.py`
- [PCAP Watcher](https://github.com/cisagov/malcolm/blob/main/shared/bin/pcap_watcher.py): `shared/bin/pcap_watcher.py`

### Processing Enrichment

- [PCAP Processor](https://github.com/cisagov/malcolm/blob/main/shared/bin/pcap_processor.py): `shared/bin/pcap_processor.py`
- [Zeek Analysis](https://github.com/cisagov/malcolm/tree/main/zeek): `zeek`
- [Suricata Analysis](https://github.com/cisagov/malcolm/tree/main/suricata): `suricata`
- [File Scanning](https://github.com/cisagov/malcolm/tree/main/filescan/python-filescan/filescan): `filescan/python-filescan/filescan`
- [Normalization Enrichment](https://github.com/cisagov/malcolm/tree/main/logstash): `logstash`
- [Asset Inventory](https://github.com/cisagov/malcolm/tree/main/netbox): `netbox`

### Data Services

- [OpenSearch](https://github.com/cisagov/malcolm/tree/main/dashboards): `dashboards`

### Analysis Interfaces

- [Arkime Sessions](https://github.com/cisagov/malcolm/tree/main/arkime): `arkime`
- [WISE Fields](https://github.com/cisagov/malcolm/blob/main/arkime/wise/source.zeeklogs.js): `arkime/wise/source.zeeklogs.js`
- [Dashboards](https://github.com/cisagov/malcolm/tree/main/dashboards): `dashboards`
- [Malcolm API](https://github.com/cisagov/malcolm/blob/main/api/project/__init__.py): `api/project/__init__.py`

### Deployment Operations

- [Installer Configuration](https://github.com/cisagov/malcolm/tree/main/scripts/installer): `scripts/installer`
- [Kiosk Controls](https://github.com/cisagov/malcolm/blob/main/malcolm-iso/kiosk/interface/routes.py): `malcolm-iso/kiosk/interface/routes.py`
- [Service Operations](https://github.com/cisagov/malcolm/blob/main/malcolm-iso/kiosk/interface/sysquery/sys_service.py): `malcolm-iso/kiosk/interface/sysquery/sys_service.py`

### Other components

- **Analyst**
- **Remote Forwarder**
- **Network Traffic**
- **Browser User**

## Connections

- Browser User → File Upload: uploads artifacts
- File Upload → Upload Watchers: saves uploads
- Remote Forwarder → Zeek Analysis: forwards logs
- Network Traffic → Zeek Analysis: captures traffic
- Network Traffic → Suricata Analysis: captures traffic
- PCAP Watcher → PCAP Processor: queues captures
- PCAP Processor → Zeek Analysis: dispatches analysis
- PCAP Processor → Suricata Analysis: dispatches analysis
- Zeek Analysis → Normalization Enrichment: emits logs
- Suricata Analysis → Normalization Enrichment: emits alerts
- Normalization Enrichment → Asset Inventory: looks up assets
- Normalization Enrichment → OpenSearch: writes events
- PCAP Processor → OpenSearch: stores session data
- PCAP Processor → File Scanning: scans extracted files
- OpenSearch → Malcolm API: serves searches
- Malcolm API → Dashboards: provides data
- OpenSearch → Dashboards: serves visualizations
- OpenSearch → Arkime Sessions: serves session data
- WISE Fields → Arkime Sessions: registers fields
- Analyst → Dashboards: explores traffic
- Analyst → Arkime Sessions: searches sessions
- Browser User → Kiosk Controls: requests controls
- Kiosk Controls → Service Operations: dispatches command