How nailo2c/deeplog works
Project Type and Purpose
• Project type: A machine-learning library + example pipeline for log-based anomaly detection.
• Purpose: Implement “DeepLog” in PyTorch for detecting and diagnosing anomalies in system logs (OpenStack example).
File Structure
Top-level:
• deeplog/
– Core library (deeplog.py defines model, training & inference API).
• example/
– preprocess.py: read raw logs, build event_id_map, generate training sequences.
– train.py: train the LSTM-based anomaly detector.
– predict.py: load a trained model, score new log streams, output anomalies.
– data/OpenStack/: raw log files + labels.
• requirements.txt: PyTorch and other Python deps.
README Insights
• Workflow steps: preprocess → train → predict → evaluate metrics.
• External data source: logpai’s LogHub (OpenStack).
• Uses Python venv, PyTorch as primary framework.
Main Components
a. Data Ingestion & Storage
– Raw logs (example/data/OpenStack/*.log)
– Anomaly labels file
b. Preprocessing Module
– example/preprocess.py uses deeplog library to:
• Tokenize events → build event_id_map (num-class)
• Window log sequences (window-size) → X, y for training
c. Training Component
– example/train.py
– deeplog.DeepLog model (an LSTM sequence predictor)
– Hyperparameters: epochs, num-class, num-candidates, window-size, local flag
– Output: serialized model checkpoint
d. Inference / Prediction Component
– example/predict.py
– Loads model checkpoint
– Streams new logs → computes anomaly scores
– Applies threshold → flags anomalies
e. Evaluation & Metrics
– Compare predicted anomalies vs. anomaly_labels.txt → Accuracy, Precision, Recall, F1
f. Core Library
– deeplog/deeplog.py: encapsulates model class, training loops, predict interface
g. External Dependencies
– PyTorch, NumPy, other entries in requirements.txt
– Data source: LogHub (OpenStack dataset)
h. Developer Environment
– Python venv, scripts, Colab notebook link
Relationships & Data Flow
1. Raw Logs → Preprocess → Prepared Dataset
2. Dataset → Train → Trained Model
3. New Logs + Model → Predict → Anomaly Flags
4. Predictions + Ground-Truth Labels → Evaluate → Metrics
Architectural Patterns & Principles
• Modular design: core library vs. example pipeline code
• ML pipeline pattern (ingest → preprocess → train → serve)
• Separation of concerns: data layer, model layer, evaluation layer
• Configurability via CLI args (hyperparameters)