# hiiamtzekean/streamsight architecture

> Architecture diagram of the GitHub repository hiiamtzekean/streamsight, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hiiamtzekean/streamsight
- Repository: https://github.com/hiiamtzekean/streamsight

## Overview

##### Project Type and Purpose
- Streamsight is a Python library aimed at offering a structured API to evaluate recommender systems. The project clearly splits responsibilities into distinct domains such as data loading, filtering/preprocessing, running recommendation algorithms (in various incarnations), evaluating their outputs via customized metrics, and managing different evaluation settings.
- The README confirms installation via pip/poetry and points out that its primary purpose is for offline evaluation with an emphasis on temporal data handling.

##### File Structure – Main Directories
- The repository is organized into several key areas:
- The “test” directory includes unit tests and integration tests, highlighting the coupling between components.
- The “examples” directory shows notebooks that demonstrate how to use the toolkit.
- Configuration and packaging files (pyproject.toml, .github workflows, etc.) indicate CI/CD and dependency management aspects.

##### Main Components to Be Represented
- Streamsight Core Library – this can be shown as the central engine.
- Algorithms: Contains various recommendation algorithm implementations (e.g., itemknn in multiple flavors, popularity-based, and random).
- Datasets: Abstracts data sources (e.g., Amazon, LastFM, Movielens, Yelp) and provides a uniform interface to feed data into the toolkit.
- Evaluators: Components that build the evaluation pipelines (both full-run and stream-based) to assess algorithm performance.
- Metrics: Implements performance metrics (like precision and recall).
- Preprocessing: Responsible for filtering data and running preprocessing routines before evaluation.
- Matrix: Builds and manages data structures (e.g., an interaction matrix) that are essential for collaborative filtering.
- Registries: Contains component registration logic to dynamically bind algorithms, metrics, and settings.
- Settings: Holds different evaluation configurations (e.g., leave_n_out, sliding window, single time point) that dictate how data is split and presented.
- Utils: Utility functions and helper scripts for logging, JSON/CSV conversion, directory operations, and more.

- Supporting Components:
- Testing Framework: The “test” directory ensuring the correctness of each module.
- Workflow/CI: The .github workflows indicating continuous integration using tools like Sphinx for generating documentation.
- Examples: Notebook demos that illustrate component usage.

##### Relationships and Data Flow
- How a typical evaluation workflow would proceed:
- Data is loaded from the Datasets module.
- Preprocessing is applied to the raw data.
- The filtered data is used to build an interaction Matrix.
- Recommendation Algorithms operate on this matrix (or directly on the processed data) to generate predictions.
- Evaluators orchestrate the pipeline by combining algorithms, selected evaluation settings (from the Settings module), and Metrics to assess performance.
- Registries manage loading of algorithm or metric classes and ensure that the correct module implementations are used.
- Showing evaluators consuming outputs from Algorithms and feeding them into the Metrics for validation and evaluation.

##### Architectural Patterns & Principles
- Modular Design: Each submodule (algorithms, datasets, evaluators, etc.) is designed to encapsulate specific functionality making it easy to extend or modify independently.
- Plug-in / Registry Pattern: The Registries module suggests that new algorithms or metrics can be added dynamically.
- Separation of Concerns: The system clearly differentiates between data handling, algorithmic processing, evaluation, and metrics calculation.
- Testable Architecture: The presence of a comprehensive “test” directory indicates strong adherence to testing best practices.

- Clear boundaries of each major component.
- Any external integrations if applicable (e.g., CI/CD pipelines under .github/workflows).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Central Core Library Node
    Core["Streamsight Core Library"]:::core

    %% Core Modules Subgraph
    subgraph "Core Modules"
        Datasets["Datasets (Data Sources)"]:::module
        Preproc["Preprocessing (Filtering)"]:::module
        Matrix["Matrix (Interaction Data)"]:::module
        Algorithms["Algorithms (Recommenders)"]:::module
        Evaluators["Evaluators (Pipeline Orchestrator)"]:::module
        Metrics["Metrics (Performance Measurement)"]:::module
        Settings["Settings (Evaluation Configuration)"]:::module
        Registries["Registries (Dynamic Binding)"]:::module
        Utils["Utils (Helper Functions)"]:::module
    end

    %% Supporting Components
    Doc["Documentation & Research"]:::support
    Test["Testing Framework"]:::support
    CI["Workflow/CI"]:::support
    Examples["Examples (Usage Notebooks)"]:::support

    %% Connections from Core Library to its Modules
    Core -->|"contains"| Datasets
    Core -->|"contains"| Preproc
    Core -->|"contains"| Matrix
    Core -->|"contains"| Algorithms
    Core -->|"contains"| Evaluators
    Core -->|"contains"| Metrics
    Core -->|"contains"| Settings
    Core -->|"contains"| Registries
    Core -->|"contains"| Utils

    %% Data Flow Chain
    Datasets -->|"to_Preprocessing"| Preproc
    Preproc -->|"to_Matrix"| Matrix
    Matrix -->|"to_Algorithms"| Algorithms
    Algorithms -->|"to_Evaluators"| Evaluators
    Evaluators -->|"to_Metrics"| Metrics
    Settings -->|"used_by_Evaluators"| Evaluators

    %% Registry Relationships
    Registries ---|"registers"| Algorithms
    Registries ---|"registers"| Metrics
    Registries ---|"registers"| Settings

    %% Supporting Components Connections
    Doc ---|"supports"| Core
    Test ---|"validates"| Core
    CI ---|"integrates_with"| Core
    Examples ---|"demonstrates"| Core

    %% Click Events
    click Core "https://github.com/hiiamtzekean/streamsight/blob/master/streamsight/__init__.py"
    click Algorithms "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/algorithms"
    click Datasets "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/datasets"
    click Evaluators "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/evaluators"
    click Metrics "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/metrics"
    click Preproc "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/preprocessing"
    click Matrix "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/matrix"
    click Registries "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/registries"
    click Settings "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/settings"
    click Utils "https://github.com/hiiamtzekean/streamsight/tree/master/streamsight/utils"
    click Doc "https://github.com/hiiamtzekean/streamsight/tree/master/docs"
    click Test "https://github.com/hiiamtzekean/streamsight/tree/master/test"
    click CI "https://github.com/hiiamtzekean/streamsight/tree/master/.github/workflows"
    click Examples "https://github.com/hiiamtzekean/streamsight/tree/master/examples"

    %% Styles
    classDef core fill:#87CEEB,stroke:#1E90FF,stroke-width:2px;
    classDef module fill:#FFEFD5,stroke:#FF8C00,stroke-width:2px;
    classDef support fill:#E6E6FA,stroke:#9370DB,stroke-width:2px;
```
