How ait-aecid/aecid-alert-aggregation works
Project Type and Purpose
• This is an open-source, Python-based command-line tool (aecid-alert-aggregation) for grouping, clustering, and merging semi-structured security alerts.
• It ingests raw alerts (from AMINER, Wazuh), preprocesses them, computes similarities, clusters them into groups, merges those groups into “meta-alerts,” and offers evaluation and sample scripts for further analysis.
Main Components
a. Orchestrator / CLI Entry Point
• aggregate.py – reads configuration, drives the entire pipeline.
• aggregate_config.py – specifies deltas, input/output paths, thresholds.
b. Data Ingestion & Preprocessing
• preprocessing/read_input.py – parses raw data formats (JSON, TXT).
• preprocessing/preprocess.py, label.py, objects.py – cleans alerts, assigns labels, creates in-memory alert objects.
c. Similarity Calculation
• similarity/similarity.py – defines pairwise similarity metrics between alerts and groups.
d. Clustering
• clustering/time_delta_group.py – groups alerts by temporal proximity.
• clustering/objects.py – data structures for groups.
e. Merging / Meta-Alert Generation
• merging/merge.py – incrementally merges groups into meta-alerts based on similarity thresholds.
• merging/objects.py – data structures for meta-alerts.
f. Output & Storage
• data/out/aggregate/meta_alerts.txt, alerts.txt – final outputs.
g. Evaluation & Samples (Optional Paths)
• evaluation/*.py – scripts for cross-validation, hierarchical clustering, MDS, noise robustness.
• samples/*.py – example usage of core modules in standalone scripts.
Relationships & Data Flow
• Raw Data Files → (read_input.py) → Alert Objects → (preprocess.py + label.py) → Preprocessed Alerts
• Preprocessed Alerts → (similarity.py) → Similarity Matrices
• Similarity + Temporal Window → (clustering/time_delta_group.py) → Alert Groups
• Alert Groups → (merging/merge.py) → Meta-Alerts
• Meta-Alerts → (file I/O) → data/out/aggregate/*.txt
• Optional: Meta-Alerts & Similarity Matrices → evaluation scripts → performance metrics/R-scripts
Architectural Patterns & Principles
• Pipeline Pattern: sequential stages of ingestion → preprocessing → similarity → clustering → merging.
• Modular Design: each phase lives in its own package (preprocessing, similarity, clustering, merging).
• Configuration-driven: aggregate_config.py centralizes all runtime parameters.