How vaibhav300503/agent works
1. This repository is a two-sided SOC pipeline: an endpoint agent (soc-agent/src) and a central server (soc-agent/server). The agent runs close to the source systems, while the server handles ingest, normalization, detection, and alert forwarding.
2. The agent’s core runtime is the agent.py entrypoint, which wires together platform-specific collectors, enrichment helpers, buffering/transport, and the SOAR/detection engine. It is packaged for Windows via PyInstaller and also has Linux install scripts and a Windows service wrapper.
3. Collection is split by data source and platform. collectors/windows.py, collectors/linux.py, collectors/network.py, and collectors/fim.py indicate separate subsystems for Windows event/security data, Linux logs, network telemetry, and file integrity monitoring. collectors/base.py likely defines common collector behavior and event normalization hooks.
4. Before data leaves the endpoint, it is enriched and sanitized. geo_enricher.py, ip_resolver.py, and sanitizer.py add context such as IP metadata and remove unsafe content. config.py, utils.py, and agent_config.yaml provide runtime settings like server URL, enabled collectors, and host identity.
5. Transport is resilient rather than fire-and-forget. The README and file layout show a buffered delivery model with local SQLite storage (agent_buffer.db) and transport.py for retrying sends over HTTPS to the SOC server. This is the main reliability boundary between endpoint collection and central processing.
6. The server side is organized as an ingest-and-process pipeline. server/api/ingest.py receives agent submissions, server/main.py starts the application, and server/migrate_db.py plus seed_owasp_rules.py manage persistence/bootstrap. The README indicates MongoDB as the main datastore and Redis as supporting infrastructure for asynchronous work.
7. Post-ingest processing is decomposed into workers rather than a single monolith. server/workers/parser_worker.py, enricher_worker.py, detector_worker.py, and thehive_worker.py suggest a staged flow: parse raw events, enrich them further, run detection rules, then open cases or alerts in TheHive for high-priority findings.
8. Detection is rule-driven and prioritized. The README describes OWASP, Windows, Linux, network, HTTP/web, authentication, and baseline rules with severity-based ordering and deduplication. That means the server’s detection layer is a rule engine over normalized log events, not a model-based analytics service.
9. The repo also contains deployment and operational boundaries. install/ has OS-specific install scripts and a soc-agent.service unit, while docs/ captures schema, rule sets, installation, and validation guidance. This makes the project easy to diagram as an agent package, a server service, and a documentation/ops layer.
10. Validation is treated as a first-class subsystem. validate_agent.py, validate_enrichment.py, cicd_validate.py, tests/, and server-side scripts provide automated checks for enrichment, detection behavior, and integration. The presence of mock_server.py and event-generation scripts shows the repo supports end-to-end testing without a live SOC backend.
11. Important external integrations shape the architecture: HTTPS API communication from agent to server, MongoDB for server-side storage, Redis for background processing, and TheHive for incident creation. The README also references a dashboard consumer, so the server is not only an alerting backend but also the data source for visualization.
12. For an architecture diagram, the clearest sequence is: endpoint sources → platform collectors → enrichment/sanitization → local buffer/transport → server ingest API → parser/enricher/detector workers → database/alerts → TheHive and dashboard consumers.
Main entry points: agent.py Agent runtime · geo_enricher.py Agent enrichers · detector_worker.py Detector worker · ingest.py Ingest API
Components 23
Endpoint agent
Endpoint-side collection, enrichment, buffering, and transport.
- Agent runtime · Main endpoint process wiring collectors, enrichment, SOAR logic, and transport.
soc-agent/src/agent.py - Collector base · Shared collector behavior and event normalization hooks.
soc-agent/src/collectors/base.py - Windows collector · Windows event and security data collection.
soc-agent/src/collectors/windows.py - Linux collector · Linux log and host telemetry collection.
soc-agent/src/collectors/linux.py - Network collector · Network telemetry collection.
soc-agent/src/collectors/network.py - FIM collector · File integrity monitoring collection.
soc-agent/src/collectors/fim.py - Agent enrichers · Geolocation, IP resolution, and payload sanitization before send.
soc-agent/src/geo_enricher.py - Agent config · Runtime configuration for collectors, host identity, and server endpoint.
soc-agent/src/config.py - Buffered transport · Local buffering and retrying HTTPS delivery to the server.
soc-agent/src/transport.py - SOAR engine · Endpoint-side detection or response logic integrated into the agent runtime.
soc-agent/src/soar_engine.py
SOC server
Central ingest, processing, detection, and alert forwarding.
- Server app · Starts the central SOC server application.
soc-agent/server/main.py - Ingest API · Receives submissions from agents.
soc-agent/server/api/ingest.py - Parser worker · Converts raw submissions into normalized records.
soc-agent/server/workers/parser_worker.py - Server enricher · Adds server-side enrichment to parsed events.
soc-agent/server/workers/enricher_worker.py - Detector worker · Runs rule-based detection and prioritization.
soc-agent/server/workers/detector_worker.py - TheHive worker · Forwards high-priority findings into TheHive.
soc-agent/server/workers/thehive_worker.py - MongoDB · Primary server-side persistence for events and detection state.
- Redis queue · Background-work infrastructure for asynchronous processing.
- Rule seed · Seeds OWASP and other detection rules into the server.
soc-agent/server/seed_owasp_rules.py - Dashboard · Downstream visualization or reporting consumer of server data.
- TheHive · External incident/case management target for alerts.
Ops & validation
Deployment, docs, packaging, and validation support.
- Deploy & docs · Install scripts, packaging, and architecture documentation.
soc-agent/docs/README.md - Validation · Automated checks, mock server, and end-to-end test helpers.
soc-agent/validate_agent.py
Connections 26
- Agent runtime → Collector base: uses · Agent runtime builds on shared collector behavior.
- Agent runtime → Windows collector: starts · Windows-specific collection path.
- Agent runtime → Linux collector: starts · Linux-specific collection path.
- Agent runtime → Network collector: starts · Network telemetry path.
- Agent runtime → FIM collector: starts · File integrity monitoring path.
- Windows collector → Agent enrichers: feeds · Collected events are enriched and sanitized before buffering.
- Linux collector → Agent enrichers: feeds · Collected events are enriched and sanitized before buffering.
- Network collector → Agent enrichers: feeds · Collected events are enriched and sanitized before buffering.
- FIM collector → Agent enrichers: feeds · Collected events are enriched and sanitized before buffering.
- Agent enrichers → Buffered transport: buffers · Enriched events are staged for resilient delivery.
- Agent config → Agent runtime: configures · Controls enabled collectors and server connection details.
- SOAR engine → Agent runtime: integrates · Endpoint policy and response engine runs within the agent.
- Buffered transport → Ingest API: sends · HTTPS delivery from endpoint to central ingest API.
- Server app → Ingest API: hosts · Application entrypoint exposes ingest service.
- Ingest API → Parser worker: queues · Raw submissions move into parsing stage.
- Parser worker → Server enricher: normalizes · Parsed records are enriched further.
- Server enricher → Detector worker: prepares · Enriched events feed detection rules.
- Detector worker → MongoDB: stores · Detection output and state are persisted.
- Detector worker → TheHive worker: escalates · High-priority findings become cases or alerts.
- TheHive worker → TheHive: opens cases · External case-management integration.
- MongoDB → Dashboard: serves · Dashboard consumes persisted SOC data.
- Rule seed → Detector worker: seeds rules · Bootstraps OWASP and detection rule sets.
- Validation → Agent runtime: verifies · Covers agent behavior and enrichment checks.
- Validation → Ingest API: tests · Supports mock-server and end-to-end validation.
- Deploy & docs → Agent runtime: packages · Build and installation paths for the endpoint agent.
- Deploy & docs → Server app: deploys · Server installation and operational docs.