How yuchen413/anomalyruler works
Project type and purpose
• This is an open‐source research pipeline (ECCV’24) for video anomaly detection that leverages vision‐to‐text and large language models (LLMs) to induce and deduct rules for detecting anomalies in surveillance datasets (SHTech, UBNormal, ped2, avenue).
• The pipeline is orchestrated by Python scripts (image2text.py, main.py, majority_smooth.py, etc.) and depends on PyTorch, Transformers, OpenAI API, and supporting libraries.
File structure
Top‐level directories & files:
• LICENSE, README.md – project metadata
• SHTech, UBNormal, avenue, ped2 – dataset folders each containing train.csv, test.csv, and subfolders of frame CSVs and descriptions
• dataset.py – generic dataset loader
• image2text.py – “Visual Perception” module (frame → text)
• generate_choices.py, llm.py, openai_api.py – LLM interaction and candidate generation
• main.py – entry for Rule Generation (induct) and Robust Reasoning (deduct)
• majority_smooth.py – Perception Smoothing module
• ram2tag.py, reasoning_test.py – evaluation / tagging utilities
• rule/ – static rule templates per dataset
• utils.py – shared utilities
• run.sh – convenience wrapper
README for additional insights
• Pipeline is split into two stages: Induction (visual perception → rule generation → rule aggregation) and Deduction (visual perception → smoothing → robust reasoning).
• Four sequential steps invoked by command‐line scripts.
• Dependencies include deep learning, LLM libraries, OpenAI API, and data‐processing tools.
a. Components
1) Data Layer
– Raw frames & metadata (CSV) per dataset
– dataset.py: abstract loader
2) Induction Stage
– Visual Perception Module (image2text.py)
– LLM Interface (openai_api.py, llm.py)
– Rule Generation (main.py --induct + rule/rule_*.txt)
– Rule Aggregation (generate_choices.py + voting logic)
3) Deduction Stage
– Visual Perception Module (image2text.py)
– Perception Smoothing (majority_smooth.py)
– Robust Reasoning (main.py --deduct)
4) Evaluation / Utilities
– reasoning_test.py, ram2tag.py, utils.py
5) External Services
– OpenAI API (text ↔ LLM calls)
b. Relationships & data flows
• Raw frames → dataset.py → image2text → text descriptions
• During induction: descriptions + rule templates → LLM → generated rules → voting → saved rule set
• During deduction: frames → image2text → smoothed descriptions → LLM reasoning using induced rules → final anomaly predictions
• Outputs: per‐frame anomaly labels written to CSVs under each dataset folder
c. Architectural patterns & principles
– Pipeline / staging architecture (induction vs. deduction)
– Modular separation: data loading, perception, rule handling, smoothing, reasoning
– External service integration via adapter (openai_api.py)
– Reusability: utils.py for common code, rule templates in rule/
d. Key technologies
– Python 3
– PyTorch, torchvision for frame processing
– Hugging Face Transformers for vision‐language models
– OpenAI API for LLM calls
– pandas, scikit‐learn for data handling and voting
Elements
• Clear labels on each module/component
• Blue for data/storage (datasets, CSVs)
• Green for processing modules (Python scripts)
• Orange for external services (OpenAI API)