How linkedin/isolation-forest works
Main Components
• Isolation Forest Engine (Scala/Spark Module):
• Represents the core implementation of the Isolation Forest algorithm.
• Includes components for model training, scoring, and persistence. Show that it leverages Spark’s ML library by extending Spark’s Estimator/Model and Pipeline APIs.
• ONNX Converter (Python Module):
• Represents the module responsible for converting a trained model into an ONNX format.
• That it interacts with files (model files and metadata) generated by the Scala module’s persistence mechanism.
• Conversion process and a sample Python inference workflow (using onnxruntime and numpy).
• External Services and Dependencies:
• Spark Cluster – representing the distributed nature of the training and model scoring.
• HDFS or Disk Storage – used for storing the trained model, which is then read by the Python ONNX converter.
• Maven Central/Gradle Build: Indicate that a build system (Gradle) and dependency management are in place for the Scala module, and pip (PyPI) for the Python converter.
Relationships and Interactions
• Data Flow for Training and Scoring:
• Data Ingestion: Start from raw data inputs (such as CSV files displayed in the tests) fed into a Spark DataFrame.
• Training Process: Show the flow through the IsolationForest training component with parameters (e.g., numEstimators, contamination, etc.) to produce an IsolationForestModel.
• Validation and Testing: Represent integration and unit tests that validate the logic.
• Model Persistence:
• After training, the model is saved (potentially on HDFS), with directories for “data” and “metadata.”
• ONNX Conversion Flow:
• Converter taking the saved model files as input and producing an ONNX file.
• Follow with an inference workflow (data load, pre-processing, scoring via ONNX Runtime).
Architectural Patterns and Design Principles
• Layered Architecture:
• Separate concerns by having dedicated modules for algorithm training (core library) and model portability (converter).
• Modularity and Extensibility:
• That the system is designed to be integrated with Spark ML pipelines and that the ONNX conversion process is easily pluggable, following open standards.
• Distributed Processing:
• Distributed nature via Spark and the use of pipelines, along with model persistence.
Technologies, Frameworks, and Libraries
• Scala, Spark (and Spark ML APIs) for distributed training and scoring.
• Gradle for build and dependency management.
• Python (PyPI package “isolation-forest-onnx”) for conversion.
• ONNX and ONNX Runtime for portable inference.
• HDFS or local file system for model saving.
• GitHub Actions for CI/CD (as shown in the .github/workflows directory).
Essential Elements
• Secondary components such as CI/CD pipelines (GitHub Actions) and testing modules if needed, though the focus should remain on the architectural flow of the algorithm and conversion process.