# veeranidhish-7/anamoly-detection-in-syslog-streams architecture

> Architecture diagram of the GitHub repository veeranidhish-7/anamoly-detection-in-syslog-streams, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/veeranidhish-7/anamoly-detection-in-syslog-streams
- Repository: https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams

## Overview

This repository is a research implementation for log-anomaly detection, comparing a paper-reproduction PCA/RPCA baseline with contributed models and thresholding methods. The main workflow evaluates precomputed labeled log features: split normal and anomalous samples, fit a detector on normal data, score held-out samples, apply a decision rule, and report cross-validation metrics. BGL preprocessing and sequence-based experiments are additional branches. The README is empty, and several modules are unsampled; links are limited to relationships shown in excerpts or the source index.

Main entry points: [`src/our_contributions/run_improvements.py`](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_improvements.py) (Improvement runner), [`src/baseline_replication/run_baseline.py`](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/baseline_replication/run_baseline.py) (Baseline runner), [`src/our_contributions/run_improvements_leak_fixed.py`](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_improvements_leak_fixed.py) (Leak-free runner), [`src/our_contributions/run_lstm_autoencoder.py`](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_lstm_autoencoder.py) (LSTM experiment)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_bgl_download["BGL download<br/>[bgl_pipeline.py]"]
  node_bgl_parse["Drain parsing<br/>[bgl_pipeline.py]"]
  node_bgl_features["BGL windows<br/>[bgl_pipeline.py]"]
  node_sequence_extract["Sequence extraction"]
  node_embedding_extract["Embedding extraction"]
end

subgraph group_baseline["Baseline evaluation"]
  node_baseline_runner["Baseline runner<br/>[run_baseline.py]"]
  node_pca_rpca["PCA / RPCA<br/>[pca_rpca.py]"]
  node_baseline_store[("Saved baseline models")]
end

subgraph group_improve["Improved detectors"]
  node_improvement_runner["Improvement runner"]
  node_leakfree_runner["Leak-free runner"]
  node_detectors["ML detectors<br/>[deep_learning.py]"]
  node_thresholds["Adaptive thresholds<br/>[thresholding.py]"]
  node_improved_store[("Saved contribution models")]
end

subgraph group_sequence["Sequence experiments"]
  node_lstm_runner["LSTM experiment"]
  node_lstm_model["LSTM autoencoder"]
end

subgraph group_research["Research analysis"]
  node_archive_eval["PCA/RPCA evaluation"]
  node_benchmark["Benchmarking<br/>[benchmark.py]"]
end

node_researcher(("Researcher"))
node_zenodo[("Zenodo dataset")]

node_researcher -->|"runs evaluation"| node_baseline_runner
node_researcher -->|"runs experiments"| node_improvement_runner
node_researcher -->|"runs experiment"| node_leakfree_runner
node_researcher -->|"runs experiment"| node_lstm_runner
node_bgl_download -->|"downloads archive"| node_zenodo
node_bgl_download -->|"extracts for parsing"| node_bgl_parse
node_bgl_parse -->|"structures logs"| node_bgl_features
node_baseline_runner -->|"fits and scores"| node_pca_rpca
node_baseline_runner -->|"loads or saves"| node_baseline_store
node_improvement_runner -->|"uses PCA"| node_pca_rpca
node_improvement_runner -->|"fits and scores"| node_detectors
node_improvement_runner -->|"computes cutoff"| node_thresholds
node_improvement_runner -->|"loads or saves"| node_improved_store
node_leakfree_runner -->|"fits and scores"| node_detectors
node_leakfree_runner -->|"computes cutoff"| node_thresholds
node_lstm_runner -->|"trains and scores"| node_lstm_model
node_lstm_runner -->|"computes cutoff"| node_thresholds

click node_bgl_download "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/bgl_pipeline.py"
click node_bgl_parse "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/bgl_pipeline.py"
click node_bgl_features "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/bgl_pipeline.py"
click node_sequence_extract "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/extract_sequences.py"
click node_embedding_extract "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/extract_embeddings.py"
click node_baseline_runner "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/baseline_replication/run_baseline.py"
click node_pca_rpca "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/baseline_replication/models/pca_rpca.py"
click node_baseline_store "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/tree/main/saved_models"
click node_archive_eval "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/archive/attempt_1_rank_collapse/evaluate_pca_rpca.py"
click node_improvement_runner "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_improvements.py"
click node_leakfree_runner "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_improvements_leak_fixed.py"
click node_detectors "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/models/deep_learning.py"
click node_thresholds "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/models/thresholding.py"
click node_improved_store "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/tree/main/saved_models_our_contributions"
click node_lstm_runner "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_lstm_autoencoder.py"
click node_lstm_model "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_lstm_autoencoder.py"
click node_benchmark "https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/benchmark.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bgl_download,node_bgl_parse,node_bgl_features,node_sequence_extract,node_embedding_extract toneBlue
class node_baseline_runner,node_pca_rpca,node_baseline_store,node_zenodo toneAmber
class node_improvement_runner,node_leakfree_runner,node_detectors,node_thresholds,node_improved_store toneMint
class node_lstm_runner,node_lstm_model toneRose
class node_archive_eval,node_benchmark,node_researcher toneIndigo
```

## Components

### Data preparation

- [BGL download](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/bgl_pipeline.py): `src/our_contributions/features/bgl_pipeline.py`
- [Drain parsing](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/bgl_pipeline.py): `src/our_contributions/features/bgl_pipeline.py`
- [BGL windows](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/bgl_pipeline.py): `src/our_contributions/features/bgl_pipeline.py`
- [Sequence extraction](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/extract_sequences.py): `src/our_contributions/features/extract_sequences.py`
- [Embedding extraction](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/features/extract_embeddings.py): `src/our_contributions/features/extract_embeddings.py`

### Baseline evaluation

- [Baseline runner](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/baseline_replication/run_baseline.py): `src/baseline_replication/run_baseline.py`
- [PCA / RPCA](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/baseline_replication/models/pca_rpca.py): `src/baseline_replication/models/pca_rpca.py`
- [Saved baseline models](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/tree/main/saved_models): `saved_models`

### Improved detectors

- [Improvement runner](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_improvements.py): `src/our_contributions/run_improvements.py`
- [Leak-free runner](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_improvements_leak_fixed.py): `src/our_contributions/run_improvements_leak_fixed.py`
- [ML detectors](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/models/deep_learning.py): `src/our_contributions/models/deep_learning.py`
- [Adaptive thresholds](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/models/thresholding.py): `src/our_contributions/models/thresholding.py`
- [Saved contribution models](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/tree/main/saved_models_our_contributions): `saved_models_our_contributions`

### Sequence experiments

- [LSTM experiment](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_lstm_autoencoder.py): `src/our_contributions/run_lstm_autoencoder.py`
- [LSTM autoencoder](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/run_lstm_autoencoder.py): `src/our_contributions/run_lstm_autoencoder.py`

### Research analysis

- [PCA/RPCA evaluation](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/archive/attempt_1_rank_collapse/evaluate_pca_rpca.py): `src/archive/attempt_1_rank_collapse/evaluate_pca_rpca.py`
- [Benchmarking](https://github.com/veeranidhish-7/anamoly-detection-in-syslog-streams/blob/main/src/our_contributions/benchmark.py): `src/our_contributions/benchmark.py`

### Other components

- **Researcher**
- **Zenodo dataset**

## Connections

- Researcher → Baseline runner: runs evaluation (evidence: `src/baseline_replication/run_baseline.py`)
- Researcher → Improvement runner: runs experiments (evidence: `src/our_contributions/run_improvements.py`)
- Researcher → Leak-free runner: runs experiment (evidence: `src/our_contributions/run_improvements_leak_fixed.py`)
- Researcher → LSTM experiment: runs experiment (evidence: `src/our_contributions/run_lstm_autoencoder.py`)
- BGL download → Zenodo dataset: downloads archive (evidence: `src/our_contributions/features/bgl_pipeline.py`)
- BGL download → Drain parsing: extracts for parsing (evidence: `src/our_contributions/features/bgl_pipeline.py`)
- Drain parsing → BGL windows: structures logs (evidence: `src/our_contributions/features/bgl_pipeline.py`)
- Baseline runner → PCA / RPCA: fits and scores (evidence: `src/baseline_replication/run_baseline.py`)
- Baseline runner → Saved baseline models: loads or saves (evidence: `src/baseline_replication/run_baseline.py`)
- Improvement runner → PCA / RPCA: uses PCA (evidence: `src/our_contributions/run_improvements.py`)
- Improvement runner → ML detectors: fits and scores (evidence: `src/our_contributions/run_improvements.py`)
- Improvement runner → Adaptive thresholds: computes cutoff (evidence: `src/our_contributions/run_improvements.py`)
- Improvement runner → Saved contribution models: loads or saves (evidence: `src/our_contributions/run_improvements.py`)
- Leak-free runner → ML detectors: fits and scores (evidence: `src/our_contributions/run_improvements_leak_fixed.py`)
- Leak-free runner → Adaptive thresholds: computes cutoff (evidence: `src/our_contributions/run_improvements_leak_fixed.py`)
- LSTM experiment → LSTM autoencoder: trains and scores (evidence: `src/our_contributions/run_lstm_autoencoder.py`)
- LSTM experiment → Adaptive thresholds: computes cutoff (evidence: `src/our_contributions/run_lstm_autoencoder.py`)
