How chiragfalor/ultimate-puppi works
Project type and purpose
• This is a research-oriented machine-learning pipeline (not a web app) for pileup removal and secondary vertex prediction at CERN. It uses Graph Neural Networks implemented in PyTorch (.pt model files) to classify particles/events.
• The README confirms the workflow: data preprocessing → model training (with hyperparameter search) → model saving → inference and analysis.
High-level architecture components
A. Data Sources and Configuration
• Raw event files (external data storage, e.g., ROOT or CSV files)
• Configuration files (home_path.txt, hyperparameter_list.txt, .slurm job script)
B. Preprocessing Layer
• data_read.py / upuppi_v0_dataset.py: read and wrap raw data
• data_rearrange.py → reorder vertices by pt
• data_process.py / data_process_new.py → transform to graph-network input format
C. Dataset and Loader
• dataset_graph_loader.py / dense_graph_loader.py: build PyTorch datasets and DataLoaders for GNNs
D. Model Definitions (models/)
• Graph-based architectures: DynamicPointTransformer, Dynamic_GATv2, various custom GCNs/transformers
• A common model interface in models/model.py or modelv2.py
E. Training and Hyperparameter Search
• multiclass_train_model.py, classification_train_model.py, emb_train_model.py, transformer_train_model.py
• hyperparameter_search.py / hyperparameter_search_multiclass.py: drive training loops over parameter grid
• loss_functions.py, helper_functions.py: shared utilities and custom losses
• Job scheduling via job1.slurm for HPC clusters
F. Model Persistence
• Models are checkpointed as .pt files (e.g., best_model.pt in models/ or root) and param_dict.pkl
G. Inference and Prediction
• load_pred_save.py: load a checkpoint, run forward passes on new data, save predictions, plot metrics
• best_model_prediction_events.py: event-level probability outputs
H. Analysis and Visualization
• analyze_MET.py, analyze_data.py, analyze_model.py, roc_pt.py, visualize_embeddings.py
• Results stored under results/ with subfolders by experiment name
I. Utilities and Scripts
• check_consistency.py, debug.py, read_hyperparameters.py, retrain_model.py
Relationships and Data Flow
• Raw Data → Preprocessing Layer → Graph Dataset → Training Module
↳ Hyperparameter Search drives multiple training runs → Model Checkpoints
• Best Checkpoint + New Data → Inference Module → Prediction Files
• Prediction Files → Analysis & Visualization → Results Directory
• Configuration files feed all stages with paths and parameter values; SLURM script orchestrates training on cluster.
Architectural patterns and principles
• Pipeline pattern: linear data flow through distinct stages
• Modular design: separate scripts for data, models, training, inference, analysis
• Reuse of utilities: helper_functions.py and loss_functions.py are shared across scripts
• Configuration-driven: all hyperparameters and paths externalized into text files