How giuliomat95/pydeeplog works
Project type and purpose
• DeepLog Trainer is a Python-based data-processing and machine-learning pipeline (“open-source tool”) designed to parse raw system logs and train three types of models (log-key anomaly detection, parameter-value anomaly detection, and workflow modeling) to detect and diagnose anomalies in those logs.
File-structure overview
• Top-level directories/files:
• data/ – sample logs & datasets
• deeplog_trainer/ – core library
– log_parser/ (Drain3 adapter, sessions)
– model/ (data preprocessing, training, evaluation, model management)
– parameter_detection/ (parallel structure for parameter anomaly)
– workflow/ (workflow FSM construction and network utilities)
• run/ – CLI entrypoints for each pipeline stage (run_deeplog, run_drain, run_log_key_detection_model, run_parameter_detection_model, run_workflow)
• config (hdfs_config.ini & others) for Drain parameters
• Dockerfile / requirements.txt / pytest tests
Main components & relationships
A. Ingestion & Configuration
• Raw Log Files (data/*.log, data/dataset.json)
• Config Loader (hdfs_config.ini or other .ini)
→ feeds parameters (similarity thresholds, session delimiters, LSTM hyperparameters) into downstream stages
B. Log-Parsing Stage
• Drain3 Parser (deeplog_trainer/log_parser/drain.py)
• Adapter (sessionizing logs, deeplog_trainer/log_parser/adapter.py)
• Session Builder (deeplog_trainer/log_parser/sessions.py)
→ outputs structured sequences of (log_key_id, parameter_vector, timestamps)
C. Log-Key Anomaly Detection Model
• Data Preprocessing (deeplog_trainer/model/data_preprocess.py)
• LSTM Training (deeplog_trainer/model/training.py, parameters: window_size, lstm_units, etc.)
• Model Manager / Evaluator (model_manager.py, model_evaluator.py)
→ produces logkey_model.h5, metrics, tensorboard logs
D. Parameter-Value Anomaly Detection Model
• Similar pipeline under deeplog_trainer/parameter_detection:
data_preprocess.py, model_evaluator.py
• LSTM forecasting on numerical parameter vectors + time deltas
→ produces log_par_model.h5, confidence intervals, metrics
E. Workflow Modeling
• Sequence clustering and FSM construction (deeplog_trainer/workflow/network.py, workflow.py)
• Detects concurrency divergences and loops
→ outputs workflows as pickle files
F. Orchestration & CLI
• run/ scripts wire together the above stages in configurable pipelines
• run_deeplog.py ties parsing, log-key model, parameter model, and workflow model into one command
G. Artifacts & External Services
• Artifacts Storage (artifacts/… for models, tensorboard dirs)
• Tensorboard for training visualization
• Docker container for reproducible environment
• Pytest for continuous integration
Architectural patterns & design principles
• Pipeline / staged ETL + ML training
• Modular, layered design (parser → model → evaluation → management)
• Pluggable components (e.g., different adapter types, thresholds)
• Separation of concerns: parsing, model training, workflow analysis
Detail level
• Interactions (e.g., how model_evaluator reads model outputs to compute metrics)
• Don’t omit the test suite – you may group all tests under a “CI / Testing” container