# yandex/rep architecture

> Architecture diagram of the GitHub repository yandex/rep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yandex/rep
- Repository: https://github.com/yandex/rep

## Overview

##### Main components
a. User Interface Layer
- Jupyter/IPython Notebooks (howto/)
- CLI entry points (Makefile, setup.py console scripts)
b. Core Library (“rep” package)
- Data subsystem (rep/data/storage.py)
- Estimators wrappers (rep/estimators/*.py)
- Meta-ML algorithms (rep/metaml/*.py)
- Reporting & plotting (rep/report/*.py, rep/plotting.py)
- Utility functions (rep/utils.py)
c. External ML Engines
- scikit-learn, XGBoost, TMVA, Theanets, Pybrain, Neurolab, MatrixNet service
d. Execution Infrastructure
- Local Python interpreter (2.x/3.x)
- Parallel cluster/executor (IPython.parallel or custom grid scripts)
e. Data Storage
- Local files, toy datasets (howto/toy_datasets), potential remote stores via rep/data/storage.py
f. CI/CD & Deployment
- Docker images (ci/Dockerfile.rep, howto/Dockerfile)
- Continuous integration (Travis CI, CircleCI configs, .travis.yml, circle.yml)
- Environment build scripts (ci/install_rep_environment.sh, environments_*.yaml)
g. Documentation & How-To Examples
- Sphinx docs (docs/)
- Jupyter notebooks (howto/)

##### Interactions and data flows
- Users launch Jupyter notebooks → invoke REP Core APIs
- REP Core → wraps calls to External ML Engines for fit/predict
- Data subsystem → loads/saves datasets for both training and reporting
- Meta-ML module → orchestrates grid-search, folding, stacking by coordinating multiple Estimators
- Reporting → consumes model outputs to generate metrics, plots, and HTML/PDF reports
- Parallel executor → handles distributed execution of Meta-ML tasks (grid-search, cross-validation)
- CI/CD pipeline → builds Docker images, runs unit tests (tests/ and rep/test/), generates docs, publishes to PyPI

##### Architectural patterns and design principles
- Modular layering: clear separation between wrappers (rep/estimators), orchestration (rep/metaml), and reporting
- Adapter/Wrapper pattern: uniform sklearn-compatible interfaces around heterogeneous ML backends
- Pipeline pattern: chaining data loading → training → evaluation → reporting
- Plug-in extensibility: easy to add new estimators or metrics via rep/estimators and rep/report

##### Elements
- Clear labels for every component and subcomponent
- Programming language/runtime (Python)
- Notebook interface (Jupyter)
- Docker containers/CI systems
- External services (e.g., MatrixNet API)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface Layer"
        direction TB
        Jupyter["Jupyter/IPython Notebooks"]:::ui
        CLI_Make["CLI Entry: Makefile"]:::ui
        CLI_Setup["CLI Entry: setup.py"]:::ui
    end

    %% Core Library
    subgraph "REP Core Library"
        direction TB
        DataStorage["Data Subsystem"]:::core
        Estimators["Estimators Wrappers"]:::core
        MetaML["Meta-ML Algorithms"]:::core
        Reporting["Reporting"]:::core
        Plotting["Plotting Utilities"]:::core
        Utils["General Utilities"]:::core
    end

    %% External ML Engines
    subgraph "External ML Engines" 
        direction TB
        SKL["scikit-learn"]:::external
        XGB["XGBoost"]:::external
        TMVA["TMVA"]:::external
        Theanets["Theanets"]:::external
        Pybrain["Pybrain"]:::external
        Neurolab["Neurolab"]:::external
        MatrixNet["MatrixNet Service"]:::external
    end

    %% Execution Infrastructure
    subgraph "Execution Infrastructure"
        direction TB
        Python["Local Python Interpreter"]:::external
        Parallel["Parallel Executor"]:::external
    end

    %% Data Storage
    subgraph "Data Storage"
        direction TB
        LocalFiles["Local Files"]:::external
        ToyData["Toy Datasets"]:::external
    end

    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment"
        direction TB
        DockerREP["Docker Image (REP)"]:::ci
        DockerExamples["Docker Image (Examples)"]:::ci
        Travis["Travis CI"]:::ci
        CircleCI["CircleCI"]:::ci
        EnvInstall["install_rep_environment.sh"]:::ci
        EnvTemplate["environment-rep-template.yaml"]:::ci
        Env2["environment-rep2.yaml"]:::ci
        Env3["environment-rep3.yaml"]:::ci
    end

    %% Documentation & Tests
    subgraph "Docs & Examples"
        direction TB
        Docs["Sphinx Documentation"]:::doc
    end
    subgraph "Tests"
        direction TB
        Tests["Unit Tests"]:::doc
        RepTests["REP Tests"]:::doc
    end

    %% Interactions
    Jupyter -->|"invoke REP Core APIs"| DataStorage
    CLI_Make -->|"invoke REP Core APIs"| DataStorage
    CLI_Setup -->|"invoke REP Core APIs"| DataStorage

    DataStorage -->|"loads/saves datasets"| LocalFiles
    DataStorage -->|"loads/saves datasets"| ToyData

    DataStorage -->|"provides data"| Estimators
    Estimators -->|".fit()/predict() calls"| SKL
    Estimators -->|"wraps"| XGB
    Estimators -->|"wraps"| TMVA
    Estimators -->|"wraps"| Theanets
    Estimators -->|"wraps"| Pybrain
    Estimators -->|"wraps"| Neurolab
    Estimators -->|"wraps"| MatrixNet

    MetaML -->|"orchestrates"| Estimators
    MetaML -->|"uses distributed execution"| Parallel
    Estimators -->|"runs on"| Python

    Reporting -->|"consumes model outputs"| Reporting
    Reporting -->|"generates plots"| Plotting
    Reporting -->|"outputs HTML/PDF"| LocalFiles

    Travis -->|"builds & tests"| DockerREP
    CircleCI -->|"builds & tests"| DockerExamples
    EnvInstall -->|"sets up environment"| DockerREP
    EnvTemplate -->|"defines environment"| DockerREP
    Env2 -->|"defines environment"| DockerREP
    Env3 -->|"defines environment"| DockerREP
    Tests -->|"CI pipeline tests"| Travis
    RepTests -->|"CI pipeline tests"| CircleCI

    Docs -->|"published by"| Travis

    %% Click events
    click Jupyter "https://github.com/yandex/rep/tree/master/howto/"
    click CLI_Make "https://github.com/yandex/rep/tree/master/Makefile"
    click CLI_Setup "https://github.com/yandex/rep/blob/master/setup.py"
    click DataStorage "https://github.com/yandex/rep/blob/master/rep/data/storage.py"
    click Estimators "https://github.com/yandex/rep/tree/master/rep/estimators/"
    click MetaML "https://github.com/yandex/rep/tree/master/rep/metaml/"
    click Reporting "https://github.com/yandex/rep/tree/master/rep/report/"
    click Plotting "https://github.com/yandex/rep/blob/master/rep/plotting.py"
    click Utils "https://github.com/yandex/rep/blob/master/rep/utils.py"
    click ToyData "https://github.com/yandex/rep/tree/master/howto/toy_datasets/"
    click DockerREP "https://github.com/yandex/rep/blob/master/ci/Dockerfile.rep"
    click DockerExamples "https://github.com/yandex/rep/tree/master/howto/Dockerfile"
    click Travis "https://github.com/yandex/rep/blob/master/.travis.yml"
    click CircleCI "https://github.com/yandex/rep/blob/master/circle.yml"
    click EnvInstall "https://github.com/yandex/rep/blob/master/ci/install_rep_environment.sh"
    click EnvTemplate "https://github.com/yandex/rep/blob/master/ci/environment-rep-template.yaml"
    click Env2 "https://github.com/yandex/rep/blob/master/ci/environment-rep2.yaml"
    click Env3 "https://github.com/yandex/rep/blob/master/ci/environment-rep3.yaml"
    click Docs "https://github.com/yandex/rep/tree/master/docs/"
    click Tests "https://github.com/yandex/rep/tree/master/tests/"
    click RepTests "https://github.com/yandex/rep/tree/master/rep/test/"

    %% Styles
    classDef ui fill:#FFF7C0,stroke:#E6B800
    classDef core fill:#CCE5FF,stroke:#3399FF
    classDef external fill:#E0FFE0,stroke:#33CC33
    classDef ci fill:#FFE0CC,stroke:#FF9933
    classDef doc fill:#F0E0FF,stroke:#9933CC

    %% Legend
    subgraph Legend
        direction TB
        UI["User Interface"]:::ui
        CORE["REP Core"]:::core
        EXT["External/Runtime"]:::external
        CI["CI/CD & Docker"]:::ci
        DOC["Docs & Tests"]:::doc
    end
```
