# ait-aecid/aecid-alert-aggregation architecture

> Architecture diagram of the GitHub repository ait-aecid/aecid-alert-aggregation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ait-aecid/aecid-alert-aggregation
- Repository: https://github.com/ait-aecid/aecid-alert-aggregation

## Overview

##### Project Type and Purpose
- This is an open-source, Python-based command-line tool (aecid-alert-aggregation) for grouping, clustering, and merging semi-structured security alerts.
- It ingests raw alerts (from AMINER, Wazuh), preprocesses them, computes similarities, clusters them into groups, merges those groups into “meta-alerts,” and offers evaluation and sample scripts for further analysis.

##### Main Components
a. Orchestrator / CLI Entry Point
- aggregate.py – reads configuration, drives the entire pipeline.
- aggregate_config.py – specifies deltas, input/output paths, thresholds.
b. Data Ingestion & Preprocessing
- preprocessing/read_input.py – parses raw data formats (JSON, TXT).
- preprocessing/preprocess.py, label.py, objects.py – cleans alerts, assigns labels, creates in-memory alert objects.
c. Similarity Calculation
- similarity/similarity.py – defines pairwise similarity metrics between alerts and groups.
d. Clustering
- clustering/time_delta_group.py – groups alerts by temporal proximity.
- clustering/objects.py – data structures for groups.
e. Merging / Meta-Alert Generation
- merging/merge.py – incrementally merges groups into meta-alerts based on similarity thresholds.
- merging/objects.py – data structures for meta-alerts.
f. Output & Storage
- data/out/aggregate/meta_alerts.txt, alerts.txt – final outputs.
g. Evaluation & Samples (Optional Paths)
- evaluation/*.py – scripts for cross-validation, hierarchical clustering, MDS, noise robustness.
- samples/*.py – example usage of core modules in standalone scripts.

##### Relationships & Data Flow
- Raw Data Files → (read_input.py) → Alert Objects → (preprocess.py + label.py) → Preprocessed Alerts
- Preprocessed Alerts → (similarity.py) → Similarity Matrices
- Similarity + Temporal Window → (clustering/time_delta_group.py) → Alert Groups
- Alert Groups → (merging/merge.py) → Meta-Alerts
- Meta-Alerts → (file I/O) → data/out/aggregate/*.txt
- Optional: Meta-Alerts & Similarity Matrices → evaluation scripts → performance metrics/R-scripts

##### Architectural Patterns & Principles
- Pipeline Pattern: sequential stages of ingestion → preprocessing → similarity → clustering → merging.
- Modular Design: each phase lives in its own package (preprocessing, similarity, clustering, merging).
- Configuration-driven: aggregate_config.py centralizes all runtime parameters.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI / Orchestrator
    subgraph "CLI / Orchestrator"
        AGG["aggregate.py"]:::script
        CONFIG["aggregate_config.py"]:::script
    end

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        READ["read_input.py"]:::module
        PREP["preprocess.py"]:::module
        LABEL["label.py"]:::module
        POBJ["objects.py"]:::module
    end

    %% Similarity Calculation
    subgraph "Similarity Calculation"
        SIM["similarity.py"]:::module
    end

    %% Clustering
    subgraph "Clustering"
        TDG["time_delta_group.py"]:::module
        COBJ["objects.py"]:::module
    end

    %% Merging / Meta-Alert Generation
    subgraph "Merging / Meta-Alert Generation"
        MERGE["merge.py"]:::module
        MOBJ["objects.py"]:::module
    end

    %% Output & Storage
    subgraph "Output & Storage"
        METAOUT["meta_alerts.txt"]:::data
        ALERTOUT["alerts.txt"]:::data
    end

    %% Evaluation Scripts
    subgraph "Evaluation Scripts"
        EV1["cross_validation.py"]:::script
        EV2["evaluate.py"]:::script
        EV3["hierarchical_clustering.py"]:::script
        EV4["mds.py"]:::script
        EV5["noise_evaluate.py"]:::script
    end

    %% Sample Usage Scripts
    subgraph "Sample Usage Scripts"
        SAMP1["sample.py"]:::script
        SAMP2["sample_group_merge.py"]:::script
        SAMP3["sample_group_similarity.py"]:::script
        SAMP4["sample_hierarchical_clustering.py"]:::script
        SAMP5["sample_merge.py"]:::script
        SAMP6["sample_similarity.py"]:::script
    end

    %% Data Flow Connections
    AGG -->|"reads config"| CONFIG
    AGG -->|"initiates ingestion"| READ
    READ -->|"Alert JSON/TXT"| PREP
    PREP -->|"Preprocessed Alerts"| SIM
    SIM -->|"Similarity Matrix"| TDG
    TDG -->|"Alert Groups"| MERGE
    MERGE -->|"Meta-Alerts"| METAOUT
    MERGE -->|"Alerts"| ALERTOUT

    %% Evaluation & Samples Integration
    SIM -->|"Similarity Matrix"| EV1
    SIM -->|"Similarity Matrix"| EV2
    MERGE -->|"Meta-Alerts"| EV3
    MERGE -->|"Meta-Alerts"| EV4
    MERGE -->|"Meta-Alerts"| EV5

    SAMP1 -->|"uses preprocessing"| PREP
    SAMP2 -->|"uses merging"| MERGE
    SAMP3 -->|"uses similarity"| SIM
    SAMP4 -->|"uses clustering"| TDG
    SAMP5 -->|"uses merging"| MERGE
    SAMP6 -->|"uses similarity"| SIM

    %% Click Events
    click AGG "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/aggregate.py"
    click CONFIG "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/aggregate_config.py"
    click READ "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/preprocessing/read_input.py"
    click PREP "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/preprocessing/preprocess.py"
    click LABEL "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/preprocessing/label.py"
    click POBJ "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/preprocessing/objects.py"
    click SIM "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/similarity/similarity.py"
    click TDG "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/clustering/time_delta_group.py"
    click COBJ "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/clustering/objects.py"
    click MERGE "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/merging/merge.py"
    click MOBJ "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/merging/objects.py"
    click METAOUT "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/data/out/aggregate/meta_alerts.txt"
    click ALERTOUT "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/data/out/sample/alerts.txt"
    click EV1 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/evaluation/cross_validation.py"
    click EV2 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/evaluation/evaluate.py"
    click EV3 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/evaluation/hierarchical_clustering.py"
    click EV4 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/evaluation/mds.py"
    click EV5 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/evaluation/noise_evaluate.py"
    click SAMP1 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/samples/sample.py"
    click SAMP2 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/samples/sample_group_merge.py"
    click SAMP3 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/samples/sample_group_similarity.py"
    click SAMP4 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/samples/sample_hierarchical_clustering.py"
    click SAMP5 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/samples/sample_merge.py"
    click SAMP6 "https://github.com/ait-aecid/aecid-alert-aggregation/blob/master/samples/sample_similarity.py"

    %% Styles
    classDef module fill:#D0E1FF,stroke:#0366d6;
    classDef data fill:#DFFFE0,stroke:#228B22;
    classDef script fill:#FFE4B5,stroke:#FF8C00;
```
