How yandex/rep works
Main components
a. User Interface Layer
• Jupyter/IPython Notebooks (howto/)
• CLI entry points (Makefile, setup.py console scripts)
b. Core Library (“rep” package)
• Data subsystem (rep/data/storage.py)
• Estimators wrappers (rep/estimators/*.py)
• Meta-ML algorithms (rep/metaml/*.py)
• Reporting & plotting (rep/report/*.py, rep/plotting.py)
• Utility functions (rep/utils.py)
c. External ML Engines
• scikit-learn, XGBoost, TMVA, Theanets, Pybrain, Neurolab, MatrixNet service
d. Execution Infrastructure
• Local Python interpreter (2.x/3.x)
• Parallel cluster/executor (IPython.parallel or custom grid scripts)
e. Data Storage
• Local files, toy datasets (howto/toy_datasets), potential remote stores via rep/data/storage.py
f. CI/CD & Deployment
• Docker images (ci/Dockerfile.rep, howto/Dockerfile)
• Continuous integration (Travis CI, CircleCI configs, .travis.yml, circle.yml)
• Environment build scripts (ci/install_rep_environment.sh, environments_*.yaml)
g. Documentation & How-To Examples
• Sphinx docs (docs/)
• Jupyter notebooks (howto/)
Interactions and data flows
• Users launch Jupyter notebooks → invoke REP Core APIs
• REP Core → wraps calls to External ML Engines for fit/predict
• Data subsystem → loads/saves datasets for both training and reporting
• Meta-ML module → orchestrates grid-search, folding, stacking by coordinating multiple Estimators
• Reporting → consumes model outputs to generate metrics, plots, and HTML/PDF reports
• Parallel executor → handles distributed execution of Meta-ML tasks (grid-search, cross-validation)
• CI/CD pipeline → builds Docker images, runs unit tests (tests/ and rep/test/), generates docs, publishes to PyPI
Architectural patterns and design principles
• Modular layering: clear separation between wrappers (rep/estimators), orchestration (rep/metaml), and reporting
• Adapter/Wrapper pattern: uniform sklearn-compatible interfaces around heterogeneous ML backends
• Pipeline pattern: chaining data loading → training → evaluation → reporting
• Plug-in extensibility: easy to add new estimators or metrics via rep/estimators and rep/report
Elements
• Clear labels for every component and subcomponent
• Programming language/runtime (Python)
• Notebook interface (Jupyter)
• Docker containers/CI systems
• External services (e.g., MatrixNet API)