# nailo2c/deeplog architecture

> Architecture diagram of the GitHub repository nailo2c/deeplog, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nailo2c/deeplog
- Repository: https://github.com/nailo2c/deeplog

## Overview

##### Project Type and Purpose
- Project type: A machine-learning library + example pipeline for log-based anomaly detection.
- Purpose: Implement “DeepLog” in PyTorch for detecting and diagnosing anomalies in system logs (OpenStack example).

##### File Structure
Top-level:
- deeplog/
– Core library (deeplog.py defines model, training & inference API).
- example/
– preprocess.py: read raw logs, build event_id_map, generate training sequences.
– train.py: train the LSTM-based anomaly detector.
– predict.py: load a trained model, score new log streams, output anomalies.
– data/OpenStack/: raw log files + labels.
- requirements.txt: PyTorch and other Python deps.

##### README Insights
- Workflow steps: preprocess → train → predict → evaluate metrics.
- External data source: logpai’s LogHub (OpenStack).
- Uses Python venv, PyTorch as primary framework.

##### Main Components
a. Data Ingestion & Storage
– Raw logs (example/data/OpenStack/*.log)
– Anomaly labels file
b. Preprocessing Module
– example/preprocess.py uses deeplog library to:
- Tokenize events → build event_id_map (num-class)
- Window log sequences (window-size) → X, y for training
c. Training Component
– example/train.py
– deeplog.DeepLog model (an LSTM sequence predictor)
– Hyperparameters: epochs, num-class, num-candidates, window-size, local flag
– Output: serialized model checkpoint
d. Inference / Prediction Component
– example/predict.py
– Loads model checkpoint
– Streams new logs → computes anomaly scores
– Applies threshold → flags anomalies
e. Evaluation & Metrics
– Compare predicted anomalies vs. anomaly_labels.txt → Accuracy, Precision, Recall, F1
f. Core Library
– deeplog/deeplog.py: encapsulates model class, training loops, predict interface
g. External Dependencies
– PyTorch, NumPy, other entries in requirements.txt
– Data source: LogHub (OpenStack dataset)
h. Developer Environment
– Python venv, scripts, Colab notebook link

##### Relationships & Data Flow
1. Raw Logs → Preprocess → Prepared Dataset
2. Dataset → Train → Trained Model
3. New Logs + Model → Predict → Anomaly Flags
4. Predictions + Ground-Truth Labels → Evaluate → Metrics

##### Architectural Patterns & Principles
- Modular design: core library vs. example pipeline code
- ML pipeline pattern (ingest → preprocess → train → serve)
- Separation of concerns: data layer, model layer, evaluation layer
- Configurability via CLI args (hyperparameters)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    loghub["LogHub (OpenStack dataset)"]:::external

    %% Data Store
    subgraph "Raw Logs & Labels"
        logs1[(openstack_normal1.log)]:::data
        logs2[(openstack_normal2.log)]:::data
        logs3[(openstack_abnormal.log)]:::data
        labels[(anomaly_labels.txt)]:::data
    end

    %% Example Pipeline
    subgraph "Example Pipeline"
        pre["Preprocessor\n(example/preprocess.py)"]:::component
        train["Trainer\n(example/train.py)"]:::component
        predict["Predictor\n(example/predict.py)"]:::component
        metrics["Evaluator\n(Accuracy, Precision, Recall, F1)"]:::component
    end

    %% Core Library
    subgraph "Core Library"
        core["deeplog.DeepLog\n(LSTM Model & API)"]:::core
        init["Package Init\n(deeplog/__init__.py)"]:::core
    end

    %% Developer Environment & Docs
    env["Developer Environment\n(venv, Colab)"]:::external
    req["requirements.txt"]:::external
    readme["README.md"]:::external

    %% Data Flows
    logs1 -->|raw logs| pre
    logs2 -->|raw logs| pre
    logs3 -->|raw logs| pre
    pre -->|"event_id_map, sequences"| train
    train -->|"model.pt checkpoint"| predict
    predict -->|"anomaly flags"| metrics
    labels -->|ground-truth| metrics

    %% Core Library Usage
    pre -->|uses| core
    train -->|uses| core
    predict -->|uses| core

    %% External and Environment
    loghub -->|source| logs1
    env -.-> pre
    env -.-> train
    env -.-> predict

    %% Click Events
    click core "https://github.com/nailo2c/deeplog/blob/master/deeplog/deeplog.py"
    click init "https://github.com/nailo2c/deeplog/blob/master/deeplog/__init__.py"
    click pre "https://github.com/nailo2c/deeplog/blob/master/example/preprocess.py"
    click train "https://github.com/nailo2c/deeplog/blob/master/example/train.py"
    click predict "https://github.com/nailo2c/deeplog/blob/master/example/predict.py"
    click logs1 "https://github.com/nailo2c/deeplog/blob/master/example/data/OpenStack/openstack_normal1.log"
    click logs2 "https://github.com/nailo2c/deeplog/blob/master/example/data/OpenStack/openstack_normal2.log"
    click logs3 "https://github.com/nailo2c/deeplog/blob/master/example/data/OpenStack/openstack_abnormal.log"
    click labels "https://github.com/nailo2c/deeplog/blob/master/example/data/OpenStack/anomaly_labels.txt"
    click req "https://github.com/nailo2c/deeplog/blob/master/requirements.txt"
    click readme "https://github.com/nailo2c/deeplog/blob/master/README.md"

    %% Styles
    classDef component fill:#AED6F1,stroke:#2E86C1,color:#1B4F72,stroke-width:1px
    classDef core fill:#FCF3CF,stroke:#B7950B,color:#7D6608,stroke-width:1px
    classDef data fill:#A9DFBF,stroke:#27AE60,color:#196F3D,stroke-width:1px
    classDef external fill:#D5D8DC,stroke:#566573,color:#2C3E50,stroke-width:1px
    classDef componentMetrics fill:#F5B7B1,stroke:#C0392B,color:#641E16,stroke-width:1px
    class metrics componentMetrics
```
