# linkedin/isolation-forest architecture

> Architecture diagram of the GitHub repository linkedin/isolation-forest, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linkedin/isolation-forest
- Repository: https://github.com/linkedin/isolation-forest

## Overview

##### Main Components
- Isolation Forest Engine (Scala/Spark Module):
- Represents the core implementation of the Isolation Forest algorithm.
- Includes components for model training, scoring, and persistence. Show that it leverages Spark’s ML library by extending Spark’s Estimator/Model and Pipeline APIs.
- ONNX Converter (Python Module):
- Represents the module responsible for converting a trained model into an ONNX format.
- That it interacts with files (model files and metadata) generated by the Scala module’s persistence mechanism.
- Conversion process and a sample Python inference workflow (using onnxruntime and numpy).
- External Services and Dependencies:
- Spark Cluster – representing the distributed nature of the training and model scoring.
- HDFS or Disk Storage – used for storing the trained model, which is then read by the Python ONNX converter.
- Maven Central/Gradle Build: Indicate that a build system (Gradle) and dependency management are in place for the Scala module, and pip (PyPI) for the Python converter.

##### Relationships and Interactions
- Data Flow for Training and Scoring:
- Data Ingestion: Start from raw data inputs (such as CSV files displayed in the tests) fed into a Spark DataFrame.
- Training Process: Show the flow through the IsolationForest training component with parameters (e.g., numEstimators, contamination, etc.) to produce an IsolationForestModel.
- Validation and Testing: Represent integration and unit tests that validate the logic.
- Model Persistence:
- After training, the model is saved (potentially on HDFS), with directories for “data” and “metadata.”
- ONNX Conversion Flow:
- Converter taking the saved model files as input and producing an ONNX file.
- Follow with an inference workflow (data load, pre-processing, scoring via ONNX Runtime).

##### Architectural Patterns and Design Principles
- Layered Architecture:
- Separate concerns by having dedicated modules for algorithm training (core library) and model portability (converter).
- Modularity and Extensibility:
- That the system is designed to be integrated with Spark ML pipelines and that the ONNX conversion process is easily pluggable, following open standards.
- Distributed Processing:
- Distributed nature via Spark and the use of pipelines, along with model persistence.

##### Technologies, Frameworks, and Libraries
- Scala, Spark (and Spark ML APIs) for distributed training and scoring.
- Gradle for build and dependency management.
- Python (PyPI package “isolation-forest-onnx”) for conversion.
- ONNX and ONNX Runtime for portable inference.
- HDFS or local file system for model saving.
- GitHub Actions for CI/CD (as shown in the .github/workflows directory).

##### Essential Elements
- Secondary components such as CI/CD pipelines (GitHub Actions) and testing modules if needed, though the focus should remain on the architectural flow of the algorithm and conversion process.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data and Services
    dataIngestion["Data Ingestion (CSV/Raw Data)"]:::external
    sparkCluster["Spark Cluster"]:::external
    gradleBuild["Gradle Build & Dependency"]:::build
    pythonPackaging["Python Packaging & Configuration"]:::python
    ciCd["CI/CD Pipeline"]:::ci

    %% Isolation Forest Engine (Scala/Spark Module)
    subgraph "Isolation Forest Engine (Scala/Spark Module)"
        engine["Isolation Forest Engine"]:::scala
        engine --> df["Spark DataFrame"]:::scala
        df --> training["IsolationForest Training Process"]:::scala
        training --> persistence["Model Persistence (HDFS/Disk)"]:::storage
        training --> coreClasses["Core Classes: IsolationForest, IsolationForestModel, IsolationTree, Nodes"]:::scala
        training --> supportUtilities["Support Utilities (BaggedPoint, SharedTrainLogic, Utils)"]:::scala
    end

    %% ONNX Converter (Python Module)
    subgraph "ONNX Converter (Python Module)"
        onnxConversion["ONNX Conversion Process"]:::python
        onnxConversion --> inference["Python Inference Workflow (ONNXRuntime)"]:::python
    end

    %% Data Flow Connections
    dataIngestion --> df
    sparkCluster --> df
    persistence --> onnxConversion
    gradleBuild --> engine
    pythonPackaging --> onnxConversion
    ciCd --> gradleBuild
    ciCd --> pythonPackaging

    %% Click Events
    click engine "https://github.com/linkedin/isolation-forest/tree/master/isolation-forest/src/main/scala/com/linkedin/relevance/isolationforest/"
    click coreClasses "https://github.com/linkedin/isolation-forest/tree/master/isolation-forest/src/main/scala/com/linkedin/relevance/isolationforest/"
    click supportUtilities "https://github.com/linkedin/isolation-forest/tree/master/isolation-forest/src/main/scala/com/linkedin/relevance/isolationforest/core/"
    click onnxConversion "https://github.com/linkedin/isolation-forest/blob/master/isolation-forest-onnx/src/isolationforestonnx/isolation_forest_converter.py"
    click pythonPackaging "https://github.com/linkedin/isolation-forest/tree/master/isolation-forest-onnx/"
    click ciCd "https://github.com/linkedin/isolation-forest/blob/master/.github/workflows/ci.yml"
    click gradleBuild "https://github.com/linkedin/isolation-forest/blob/master/build.gradle"

    %% Styles
    classDef scala fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef python fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef external fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef storage fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef build fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef ci fill:#FDEBD0,stroke:#D35400,stroke-width:2px;
```
