# mentat-is/gulp architecture

> Architecture diagram of the GitHub repository mentat-is/gulp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mentat-is/gulp
- Repository: https://github.com/mentat-is/gulp

## Overview

1. **What this repo is**
   - `gulp` is a Python backend service for incident-response log processing.
   - The main runtime is the `src/gulp` package; the web UI is a separate project (`gulpui-web`) and is not part of this repo.
   - The backend centers on ingesting many log/data formats, normalizing them into ECS-like documents, storing/searching them in OpenSearch, and coordinating collaborative analysis in PostgreSQL.

2. **Process entrypoint and configuration**
   - `src/gulp/__main__.py` and `src/gulp/gulp.py` are the application entrypoints.
   - `src/gulp/config.py` plus `src/gulp/gulp_cfg_template.json` define the local config model.
   - The runtime is designed around a working directory (`GULP_WORKING_DIR`) that holds config, certs, temp uploads, and optional override plugins/mappings.

3. **Core backend boundary**
   - `src/gulp/api/server_api.py`, `collab_api.py`, `opensearch_api.py`, `ws_api.py`, `redis_api.py`, `s3_api.py`, and `prometheus_api.py` are the top-level integration surfaces.
   - These modules expose the backend as REST and WebSocket services and wrap the external systems the engine depends on.
   - The server-side domain logic lives under `src/gulp/api/server/`.

4. **Ingestion pipeline**
   - Ingest is handled by `src/gulp/api/server/ingest.py`, with supporting storage and utility code in `storage.py`, `db.py`, and `server_utils.py`.
   - The engine uses multiprocessing (`src/gulp/process.py`) to parallelize parsing and indexing, which matters for diagramming because ingestion is not a single-threaded request/response path.
   - Ingestion flows from uploaded/raw source data into plugin parsers, then into normalized records, then into OpenSearch and related metadata stores.

5. **Plugin architecture**
   - `src/gulp/plugin.py` is the plugin abstraction.
   - `src/gulp/plugins/` contains format-specific parsers and enrichers: Apache, IIS, Zeek, Suricata, Windows EVTX/REG, PCAP, SQLite, JSON, EML, syslog, pfSense, TeamViewer, and others.
   - Some plugins are “stacked” or layered parsers, meaning one source can be parsed by multiple stages or embedded formats.
   - `src/gulp/plugins/extension/` contains higher-level extensions such as AI assistant, bridge manager, OTel receiver, and shared-object/plugin communication.

6. **Mapping and normalization layer**
   - `src/gulp/api/mapping/` and `src/gulp/mapping_files/` define how parsed source fields map to the target document model.
   - The repo ships many source-specific mapping files for Windows, Zeek, Suricata, PCAP, OTel, Wazuh, Apache, Chrome, and forensic tools.
   - ECS compatibility is explicit: the system normalizes diverse inputs into a common schema before indexing.

7. **Search and query path**
   - Search/query behavior is implemented in `src/gulp/api/server/query.py` and `src/gulp/api/opensearch_api.py`.
   - `src/gulp/api/opensearch/` includes query filters, Sigma support, and OpenSearch-related structures.
   - The architecture supports both direct queries and Sigma-rule-driven searches, so the diagram should show OpenSearch as the primary search engine and Sigma as a query translation layer.

8. **Collaboration subsystem**
   - `src/gulp/api/collab/` models incident collaboration objects: notes, highlights, links, users, groups, sessions, sources, and operations.
   - `collab_migrate/` contains versioned migration scripts, indicating a schema-managed PostgreSQL-backed collaboration database.
   - This subsystem is separate from search/indexing and holds analyst-facing annotations and operational context.

9. **Realtime coordination and multi-instance behavior**
   - `src/gulp/api/server/ws.py` and `src/gulp/api/ws_api.py` provide WebSocket-based notifications.
   - `src/gulp/api/redis_api.py` and the integration tests around Redis ownership/backpressure indicate Redis is used for coordination across workers/nodes, not just caching.
   - The repo explicitly supports multi-instance routing and broadcast semantics, so the architecture is distributed rather than single-node only.

10. **External services and storage**
   - OpenSearch is the indexed document store and query backend.
   - PostgreSQL stores collaboration data and user/ACL-related state.
   - Redis is used for distributed coordination and task/ownership semantics.
   - MinIO/S3-compatible storage is used for larger binary artifacts or plugin needs.
   - These services are all wired through environment overrides and Docker Compose for local/production-style deployment.

11. **Security and access control**
   - `src/gulp/api/server/object_acl.py`, `user.py`, `user_group.py`, and `auth`-related test coverage show that authorization is part of the backend boundary.
   - SSL/TLS is supported for Gulp itself and for its external service connections, with certificates expected under the working directory.

12. **Observability and operations**
   - `prometheus.yml`, `prometheus_alerts.yml`, and `docs/observability.md` show first-class metrics/alerting support.
   - There are docs and compose profiles for admin tooling around PostgreSQL, OpenSearch, Redis, and MinIO.
   - This repo also includes production/runbook docs and a Grafana dashboard export, which matter for deployment architecture but are not part of the request path.

13. **Repo structure beyond runtime**
   - `tests/` contains extensive unit, integration, e2e, and multi-instance tests; these are valuable for understanding the intended boundaries.
   - `samples/` provides representative inputs for many supported formats and is useful for visualizing ingest sources.
   - `docs/architecture.md` and `docs/plugins_and_mapping.md` are the most relevant companion documents if you need deeper subsystem-level detail.

Main entry points: [`src/gulp/api/server_api.py`](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server_api.py) (Server API), [`src/gulp/api/server/ingest.py`](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/ingest.py) (Ingest), [`src/gulp/gulp.py`](https://github.com/mentat-is/gulp/blob/master/src/gulp/gulp.py) (Gulp app), [`src/gulp/api/collab_api.py`](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/collab_api.py) (Collab API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_entry(("Main<br/>entrypoint<br/>[__main__.py]"))
  node_app["Gulp app<br/>app bootstrap<br/>[gulp.py]"]
  node_config["Config<br/>[config.py]"]
end

subgraph group_api["API & engine"]
  node_server_api["Server API<br/>service facade<br/>[server_api.py]"]
  node_ws_api["WS API<br/>realtime facade<br/>[ws_api.py]"]
  node_opensearch_api["Search API<br/>search facade<br/>[opensearch_api.py]"]
  node_redis_api["Redis API<br/>coordination facade<br/>[redis_api.py]"]
  node_s3_api["S3 API<br/>object storage facade<br/>[s3_api.py]"]
  node_prom_api["Metrics API<br/>observability facade<br/>[prometheus_api.py]"]
end

subgraph group_ingest["Ingest pipeline"]
  node_ingest["Ingest<br/>pipeline orchestrator<br/>[ingest.py]"]
  node_storage[("Storage<br/>[storage.py]")]
  node_db[("Server DB<br/>ingest metadata<br/>[db.py]")]
end

subgraph group_query["Search path"]
  node_query["Query<br/>[query.py]"]
  node_os_query["OpenSearch query<br/>query layer<br/>[filters.py]"]
  node_sigma["Sigma<br/>rule translation<br/>[sigma.py]"]
end

subgraph group_collab["Collaboration"]
  node_collab_api["Collab API<br/>collaboration facade<br/>[collab_api.py]"]
  node_collab_domain["Collab model<br/>collaboration domain<br/>[structs.py]"]
  node_migrations["Migrations<br/>[migrate.py]"]
end

subgraph group_plugins["Plugins"]
  node_plugin["Plugin API<br/>plugin abstraction<br/>[plugin.py]"]
  node_formats["Format parsers<br/>[pcap.py]"]
  node_extensions["Extensions<br/>[bridge_manager.py]"]
end

subgraph group_mapping["Mapping"]
  node_mapping_api["Mapping API<br/>mapping layer<br/>[mapping_utils.py]"]
  node_ecs_template["ECS template<br/>schema template<br/>[ecs_template.json]"]
  node_source_maps["Source maps<br/>mapping files<br/>[zeek.json]"]
end

subgraph group_ops["Operations"]
  node_observability["Observability<br/>ops assets<br/>[prometheus.yml]"]
end

node_entry -->|"starts"| node_app
node_app -->|"loads"| node_config
node_app -->|"wires"| node_server_api
node_app -->|"wires"| node_ws_api
node_server_api -->|"ingest"| node_ingest
node_server_api -->|"search"| node_query
node_server_api -->|"collab"| node_collab_api
node_server_api -->|"metrics"| node_prom_api
node_ws_api -->|"coordination"| node_redis_api
node_ws_api -->|"events"| node_collab_api
node_ingest -->|"parses with"| node_plugin
node_ingest -->|"normalizes via"| node_mapping_api
node_ingest -->|"stores"| node_storage
node_ingest -->|"records"| node_db
node_ingest -->|"indexes"| node_opensearch_api
node_plugin -->|"implements"| node_formats
node_plugin -->|"extends"| node_extensions
node_mapping_api -->|"uses"| node_ecs_template
node_mapping_api -->|"loads"| node_source_maps
node_query -->|"executes"| node_opensearch_api
node_query -->|"translates"| node_sigma
node_sigma -->|"targets"| node_opensearch_api
node_collab_api -->|"models"| node_collab_domain
node_collab_api -->|"schema"| node_migrations
node_collab_domain -->|"persists"| node_db
node_server_api -->|"routes"| node_redis_api
node_server_api -->|"artifacts"| node_s3_api
node_server_api -->|"emits"| node_observability

click node_entry "https://github.com/mentat-is/gulp/blob/master/src/gulp/__main__.py"
click node_app "https://github.com/mentat-is/gulp/blob/master/src/gulp/gulp.py"
click node_config "https://github.com/mentat-is/gulp/blob/master/src/gulp/config.py"
click node_server_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server_api.py"
click node_ws_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/ws_api.py"
click node_opensearch_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/opensearch_api.py"
click node_redis_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/redis_api.py"
click node_s3_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/s3_api.py"
click node_prom_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/prometheus_api.py"
click node_ingest "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/ingest.py"
click node_storage "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/storage.py"
click node_db "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/db.py"
click node_query "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/query.py"
click node_os_query "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/opensearch/filters.py"
click node_sigma "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/opensearch/sigma.py"
click node_collab_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/collab_api.py"
click node_collab_domain "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/collab/structs.py"
click node_migrations "https://github.com/mentat-is/gulp/blob/master/collab_migrate/v1.7.601/migrate.py"
click node_plugin "https://github.com/mentat-is/gulp/blob/master/src/gulp/plugin.py"
click node_formats "https://github.com/mentat-is/gulp/blob/master/src/gulp/plugins/pcap.py"
click node_extensions "https://github.com/mentat-is/gulp/blob/master/src/gulp/plugins/extension/bridge_manager.py"
click node_mapping_api "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/mapping/mapping_utils.py"
click node_ecs_template "https://github.com/mentat-is/gulp/blob/master/src/gulp/api/mapping/templates/ecs_template.json"
click node_source_maps "https://github.com/mentat-is/gulp/blob/master/src/gulp/mapping_files/zeek.json"
click node_observability "https://github.com/mentat-is/gulp/blob/master/prometheus.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_app,node_config,node_mapping_api,node_ecs_template,node_source_maps toneBlue
class node_server_api,node_ws_api,node_opensearch_api,node_redis_api,node_s3_api,node_prom_api,node_observability toneAmber
class node_ingest,node_storage,node_db toneMint
class node_query,node_os_query,node_sigma toneRose
class node_collab_api,node_collab_domain,node_migrations toneIndigo
class node_plugin,node_formats,node_extensions toneTeal
```

## Components

### Runtime

Application entrypoints, configuration, and process orchestration.

- [Main](https://github.com/mentat-is/gulp/blob/master/src/gulp/__main__.py): entrypoint · Package entrypoint for launching the service. · `src/gulp/__main__.py`
- [Gulp app](https://github.com/mentat-is/gulp/blob/master/src/gulp/gulp.py): app bootstrap · Application bootstrap and top-level wiring. · `src/gulp/gulp.py`
- [Config](https://github.com/mentat-is/gulp/blob/master/src/gulp/config.py): runtime config · Local configuration model and working-directory settings. · `src/gulp/config.py`

### API & engine

Backend service surfaces and server-side domain logic.

- [Server API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server_api.py): service facade · Primary backend REST surface. · `src/gulp/api/server_api.py`
- [WS API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/ws_api.py): realtime facade · WebSocket surface for notifications and coordination. · `src/gulp/api/ws_api.py`
- [Search API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/opensearch_api.py): search facade · OpenSearch integration and search backend wrapper. · `src/gulp/api/opensearch_api.py`
- [Redis API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/redis_api.py): coordination facade · Distributed ownership, routing, and backpressure coordination. · `src/gulp/api/redis_api.py`
- [S3 API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/s3_api.py): object storage facade · MinIO/S3-compatible artifact storage wrapper. · `src/gulp/api/s3_api.py`
- [Metrics API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/prometheus_api.py): observability facade · Prometheus metrics and operational instrumentation surface. · `src/gulp/api/prometheus_api.py`

### Ingest pipeline

Parsing, normalization, storage, and indexing flow.

- [Ingest](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/ingest.py): pipeline orchestrator · Upload, parse, normalize, and index pipeline. · `src/gulp/api/server/ingest.py`
- [Storage](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/storage.py): ingest storage · Artifact and chunk storage logic used during ingest. · `src/gulp/api/server/storage.py`
- [Server DB](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/db.py): ingest metadata · Server-side persistence and ingest state handling. · `src/gulp/api/server/db.py`

### Search path

OpenSearch query handling and Sigma translation.

- [Query](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/server/query.py): query engine · Search execution and query routing logic. · `src/gulp/api/server/query.py`
- [OpenSearch query](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/opensearch/filters.py): query layer · OpenSearch-specific filters, structures, and Sigma translation. · `src/gulp/api/opensearch/filters.py`
- [Sigma](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/opensearch/sigma.py): rule translation · Sigma rule support for search translation. · `src/gulp/api/opensearch/sigma.py`

### Collaboration

PostgreSQL-backed analyst state and shared context.

- [Collab API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/collab_api.py): collaboration facade · REST surface for collaborative incident state. · `src/gulp/api/collab_api.py`
- [Collab model](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/collab/structs.py): collaboration domain · Notes, highlights, links, users, groups, sessions, and operations. · `src/gulp/api/collab/structs.py`
- [Migrations](https://github.com/mentat-is/gulp/blob/master/collab_migrate/v1.7.601/migrate.py): schema migrations · Versioned PostgreSQL collaboration database migrations. · `collab_migrate/v1.7.601/migrate.py`

### Plugins

Format parsers, enrichers, and extensions.

- [Plugin API](https://github.com/mentat-is/gulp/blob/master/src/gulp/plugin.py): plugin abstraction · Plugin contract for parsers and enrichers. · `src/gulp/plugin.py`
- [Format parsers](https://github.com/mentat-is/gulp/blob/master/src/gulp/plugins/pcap.py): parsers · Representative file-format and log parsers. · `src/gulp/plugins/pcap.py`
- [Extensions](https://github.com/mentat-is/gulp/blob/master/src/gulp/plugins/extension/bridge_manager.py): plugin extensions · Higher-level plugin extensions and integration helpers. · `src/gulp/plugins/extension/bridge_manager.py`

### Mapping

Schema templates and source-to-ECS normalization.

- [Mapping API](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/mapping/mapping_utils.py): mapping layer · Mapping utilities and document model helpers. · `src/gulp/api/mapping/mapping_utils.py`
- [ECS template](https://github.com/mentat-is/gulp/blob/master/src/gulp/api/mapping/templates/ecs_template.json): schema template · ECS-compatible document template and index mapping basis. · `src/gulp/api/mapping/templates/ecs_template.json`
- [Source maps](https://github.com/mentat-is/gulp/blob/master/src/gulp/mapping_files/zeek.json): mapping files · Source-specific normalization maps for supported inputs. · `src/gulp/mapping_files/zeek.json`

### Operations

Observability and deployment support assets.

- [Observability](https://github.com/mentat-is/gulp/blob/master/prometheus.yml): ops assets · Prometheus rules, dashboard export, and operational docs. · `prometheus.yml`

## Connections

- Main → Gulp app: starts · Module entrypoint launches the application bootstrap.
- Gulp app → Config: loads · Bootstraps runtime settings and working-directory config.
- Gulp app → Server API: wires · Creates the backend service surface.
- Gulp app → WS API: wires · Enables websocket notifications and coordination.
- Server API → Ingest: ingest · Routes upload and processing requests into the ingest pipeline.
- Server API → Query: search · Serves direct search and retrieval requests.
- Server API → Collab API: collab · Exposes collaboration endpoints.
- Server API → Metrics API: metrics · Exposes operational metrics.
- WS API → Redis API: coordination · Uses Redis for multi-instance notification routing.
- WS API → Collab API: events · Broadcasts collaboration updates.
- Ingest → Plugin API: parses with · Delegates raw source handling to plugins.
- Ingest → Mapping API: normalizes via · Maps parsed fields into the common document model.
- Ingest → Storage: stores · Persists artifacts and ingest intermediates.
- Ingest → Server DB: records · Tracks ingest state and metadata.
- Ingest → Search API: indexes · Pushes normalized documents into OpenSearch.
- Plugin API → Format parsers: implements · Concrete log and file parsers.
- Plugin API → Extensions: extends · Higher-level plugin integrations and helpers.
- Mapping API → ECS template: uses · Builds normalized documents from ECS-compatible templates.
- Mapping API → Source maps: loads · Applies source-specific mapping definitions.
- Query → Search API: executes · Runs searches against the OpenSearch backend.
- Query → Sigma: translates · Converts Sigma rules into backend queries.
- Sigma → Search API: targets · Produces OpenSearch-compatible queries.
- Collab API → Collab model: models · Uses collaboration entities and operations.
- Collab API → Migrations: schema · Depends on versioned PostgreSQL migrations.
- Collab model → Server DB: persists · Stores collaboration state in PostgreSQL.
- Server API → Redis API: routes · Uses distributed ownership and routing coordination.
- Server API → S3 API: artifacts · Accesses S3-compatible binary storage.
- Server API → Observability: emits · Feeds metrics and operational monitoring.
