How coreruleset/machine-learning-integration-plugin works
Project Type and Purpose
• This is an open-source WAF (Web Application Firewall) plugin for the OWASP ModSecurity Core Rule Set (CRS) that adds a machine-learning decision layer.
• It consists of three logical parts:
• Plugin rules (.conf) loaded into CRS
• A Lua “client” script that fires on each request
• A Flask-based micro-service (ml_model_server) that computes an “anomaly score”
File Structure and Architectural Patterns
• Top-level directories:
• plugin/ : holds the ModSecurity rule files and the Lua script
• ml_model_server/ : the Python Flask application stub
• tests/ : YAML test data and a Python test harness
• Configuration files in plugin/ define thresholds, plugin mode, and the ml_server_url.
• The system follows a lightweight microservice pattern:
• The WAF (CRS) + Lua script form the entry point.
• The Flask server is a separate service, reachable via HTTP POST.
• No database is in play; the “model” is a stub within placeholder.py, but extensibility points exist under ml_model_server/saved_models.
Key Components
• External Client or Browser — initiates HTTP request
• CRS WAF Engine — loads machine-learning-before.conf and machine-learning-after.conf
• Lua Script (machine-learning-client.lua) — invoked by SecRuleScript
• HTTP Transport — POST from Lua -> Flask server
• Flask Server (ml_model_server/placeholder.py) — extracts parameters, invokes ML model stub or real model in saved_models
• ML Model Stub / Real Model — returns anomaly_score and status (0 or 1)
• CRS Decision Rules — examine inbound_ml_status to allow/block
• Optional: Test Harness (tests/) illustrating how to validate behavior
Relationships and Data Flow
• From WAF → Lua Script (label: SecRuleScript)
• Lua → Flask Server (label: HTTP POST with request metadata)
• Flask → ML Model (inside same process or separate sub-component)
• ML Model → Flask (returns anomaly_score → HTTP status code 200 or 401)
• Flask → Lua → WAF (inbound_ml_status=0|1)
• WAF → Client (block or pass)
Architectural Patterns and Design Principles
• Single Responsibility: separate WAF rules, request transformation (Lua), and ML scoring (Flask).
• Configuration-driven behavior: all thresholds and modes live in .conf files.
• Extensibility: ML model folder “saved_models” can be populated with real artifacts without changing core logic.
• Testability: standalone tests under tests/ use YAML fixtures and datavalidation_test.py.
Must-Have Elements
• Clear, human-readable labels on every component.
• Annotations for configuration files and thresholds to highlight extensibility points.