How mentat-is/gulp works
1. What this repo is
- gulp is a Python backend service for incident-response log processing.
- The main runtime is the src/gulp package; the web UI is a separate project (gulpui-web) and is not part of this repo.
- The backend centers on ingesting many log/data formats, normalizing them into ECS-like documents, storing/searching them in OpenSearch, and coordinating collaborative analysis in PostgreSQL.
2. Process entrypoint and configuration
- src/gulp/__main__.py and src/gulp/gulp.py are the application entrypoints.
- src/gulp/config.py plus src/gulp/gulp_cfg_template.json define the local config model.
- The runtime is designed around a working directory (GULP_WORKING_DIR) that holds config, certs, temp uploads, and optional override plugins/mappings.
3. Core backend boundary
- src/gulp/api/server_api.py, collab_api.py, opensearch_api.py, ws_api.py, redis_api.py, s3_api.py, and prometheus_api.py are the top-level integration surfaces.
- These modules expose the backend as REST and WebSocket services and wrap the external systems the engine depends on.
- The server-side domain logic lives under src/gulp/api/server/.
4. Ingestion pipeline
- Ingest is handled by src/gulp/api/server/ingest.py, with supporting storage and utility code in storage.py, db.py, and server_utils.py.
- The engine uses multiprocessing (src/gulp/process.py) to parallelize parsing and indexing, which matters for diagramming because ingestion is not a single-threaded request/response path.
- Ingestion flows from uploaded/raw source data into plugin parsers, then into normalized records, then into OpenSearch and related metadata stores.
5. Plugin architecture
- src/gulp/plugin.py is the plugin abstraction.
- src/gulp/plugins/ contains format-specific parsers and enrichers: Apache, IIS, Zeek, Suricata, Windows EVTX/REG, PCAP, SQLite, JSON, EML, syslog, pfSense, TeamViewer, and others.
- Some plugins are “stacked” or layered parsers, meaning one source can be parsed by multiple stages or embedded formats.
- src/gulp/plugins/extension/ contains higher-level extensions such as AI assistant, bridge manager, OTel receiver, and shared-object/plugin communication.
6. Mapping and normalization layer
- src/gulp/api/mapping/ and src/gulp/mapping_files/ define how parsed source fields map to the target document model.
- The repo ships many source-specific mapping files for Windows, Zeek, Suricata, PCAP, OTel, Wazuh, Apache, Chrome, and forensic tools.
- ECS compatibility is explicit: the system normalizes diverse inputs into a common schema before indexing.
7. Search and query path
- Search/query behavior is implemented in src/gulp/api/server/query.py and src/gulp/api/opensearch_api.py.
- src/gulp/api/opensearch/ includes query filters, Sigma support, and OpenSearch-related structures.
- The architecture supports both direct queries and Sigma-rule-driven searches, so the diagram should show OpenSearch as the primary search engine and Sigma as a query translation layer.
8. Collaboration subsystem
- src/gulp/api/collab/ models incident collaboration objects: notes, highlights, links, users, groups, sessions, sources, and operations.
- collab_migrate/ contains versioned migration scripts, indicating a schema-managed PostgreSQL-backed collaboration database.
- This subsystem is separate from search/indexing and holds analyst-facing annotations and operational context.
9. Realtime coordination and multi-instance behavior
- src/gulp/api/server/ws.py and src/gulp/api/ws_api.py provide WebSocket-based notifications.
- src/gulp/api/redis_api.py and the integration tests around Redis ownership/backpressure indicate Redis is used for coordination across workers/nodes, not just caching.
- The repo explicitly supports multi-instance routing and broadcast semantics, so the architecture is distributed rather than single-node only.
10. External services and storage
- OpenSearch is the indexed document store and query backend.
- PostgreSQL stores collaboration data and user/ACL-related state.
- Redis is used for distributed coordination and task/ownership semantics.
- MinIO/S3-compatible storage is used for larger binary artifacts or plugin needs.
- These services are all wired through environment overrides and Docker Compose for local/production-style deployment.
11. Security and access control
- src/gulp/api/server/object_acl.py, user.py, user_group.py, and auth-related test coverage show that authorization is part of the backend boundary.
- SSL/TLS is supported for Gulp itself and for its external service connections, with certificates expected under the working directory.
12. Observability and operations
- prometheus.yml, prometheus_alerts.yml, and docs/observability.md show first-class metrics/alerting support.
- There are docs and compose profiles for admin tooling around PostgreSQL, OpenSearch, Redis, and MinIO.
- This repo also includes production/runbook docs and a Grafana dashboard export, which matter for deployment architecture but are not part of the request path.
13. Repo structure beyond runtime
- tests/ contains extensive unit, integration, e2e, and multi-instance tests; these are valuable for understanding the intended boundaries.
- samples/ provides representative inputs for many supported formats and is useful for visualizing ingest sources.
- docs/architecture.md and docs/plugins_and_mapping.md are the most relevant companion documents if you need deeper subsystem-level detail.
Main entry points: server_api.py Server API · ingest.py Ingest · gulp.py Gulp app · collab_api.py Collab API
Components 25
Runtime
Application entrypoints, configuration, and process orchestration.
API & engine
Backend service surfaces and server-side domain logic.
- Server API · Primary backend REST surface.
src/gulp/api/server_api.py - WS API · WebSocket surface for notifications and coordination.
src/gulp/api/ws_api.py - Search API · OpenSearch integration and search backend wrapper.
src/gulp/api/opensearch_api.py - Redis API · Distributed ownership, routing, and backpressure coordination.
src/gulp/api/redis_api.py - S3 API · MinIO/S3-compatible artifact storage wrapper.
src/gulp/api/s3_api.py - Metrics API · Prometheus metrics and operational instrumentation surface.
src/gulp/api/prometheus_api.py
Ingest pipeline
Parsing, normalization, storage, and indexing flow.
Search path
OpenSearch query handling and Sigma translation.
- Query · Search execution and query routing logic.
src/gulp/api/server/query.py - OpenSearch query · OpenSearch-specific filters, structures, and Sigma translation.
src/gulp/api/opensearch/filters.py - Sigma · Sigma rule support for search translation.
src/gulp/api/opensearch/sigma.py
Collaboration
PostgreSQL-backed analyst state and shared context.
- Collab API · REST surface for collaborative incident state.
src/gulp/api/collab_api.py - Collab model · Notes, highlights, links, users, groups, sessions, and operations.
src/gulp/api/collab/structs.py - Migrations · Versioned PostgreSQL collaboration database migrations.
collab_migrate/v1.7.601/migrate.py
Plugins
Format parsers, enrichers, and extensions.
- Plugin API · Plugin contract for parsers and enrichers.
src/gulp/plugin.py - Format parsers · Representative file-format and log parsers.
src/gulp/plugins/pcap.py - Extensions · Higher-level plugin extensions and integration helpers.
src/gulp/plugins/extension/bridge_manager.py
Mapping
Schema templates and source-to-ECS normalization.
- Mapping API · Mapping utilities and document model helpers.
src/gulp/api/mapping/mapping_utils.py - ECS template · ECS-compatible document template and index mapping basis.
src/gulp/api/mapping/templates/ecs_template.json - Source maps · Source-specific normalization maps for supported inputs.
src/gulp/mapping_files/zeek.json
Operations
Observability and deployment support assets.
- Observability · Prometheus rules, dashboard export, and operational docs.
prometheus.yml
Connections 28
- Main → Gulp app: starts · Module entrypoint launches the application bootstrap.
- Gulp app → Config: loads · Bootstraps runtime settings and working-directory config.
- Gulp app → Server API: wires · Creates the backend service surface.
- Gulp app → WS API: wires · Enables websocket notifications and coordination.
- Server API → Ingest: ingest · Routes upload and processing requests into the ingest pipeline.
- Server API → Query: search · Serves direct search and retrieval requests.
- Server API → Collab API: collab · Exposes collaboration endpoints.
- Server API → Metrics API: metrics · Exposes operational metrics.
- WS API → Redis API: coordination · Uses Redis for multi-instance notification routing.
- WS API → Collab API: events · Broadcasts collaboration updates.
- Ingest → Plugin API: parses with · Delegates raw source handling to plugins.
- Ingest → Mapping API: normalizes via · Maps parsed fields into the common document model.
- Ingest → Storage: stores · Persists artifacts and ingest intermediates.
- Ingest → Server DB: records · Tracks ingest state and metadata.
- Ingest → Search API: indexes · Pushes normalized documents into OpenSearch.
- Plugin API → Format parsers: implements · Concrete log and file parsers.
- Plugin API → Extensions: extends · Higher-level plugin integrations and helpers.
- Mapping API → ECS template: uses · Builds normalized documents from ECS-compatible templates.
- Mapping API → Source maps: loads · Applies source-specific mapping definitions.
- Query → Search API: executes · Runs searches against the OpenSearch backend.
- Query → Sigma: translates · Converts Sigma rules into backend queries.
- Sigma → Search API: targets · Produces OpenSearch-compatible queries.
- Collab API → Collab model: models · Uses collaboration entities and operations.
- Collab API → Migrations: schema · Depends on versioned PostgreSQL migrations.
- Collab model → Server DB: persists · Stores collaboration state in PostgreSQL.
- Server API → Redis API: routes · Uses distributed ownership and routing coordination.
- Server API → S3 API: artifacts · Accesses S3-compatible binary storage.
- Server API → Observability: emits · Feeds metrics and operational monitoring.