# akochepasov/statmoments architecture

> Architecture diagram of the GitHub repository akochepasov/statmoments, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akochepasov/statmoments
- Repository: https://github.com/akochepasov/statmoments

## Overview

##### Project Type and Purpose
- This is an open-source high-performance Python library focused on computing univariate and bivariate statistical moments and t-statistics in a single streaming pass.
- The goal is to efficiently process large waveform datasets (thousands of sample points per waveform) in real-time, with an emphasis on optimizing memory usage and taking advantage of native BLAS implementations and even GPU acceleration (via the nvmath-python package).

##### File Structure
- Top-level files include standard configuration (Dockerfile, .env, etc.), build scripts (setup.py, pyproject.toml), and CI-related files (.github, .codecov.yml).
- The key directory is "statmoments" which contains:
- Core modules: __init__.py, univar.py and bivar.py for univariate and bivariate operations.
- Native/Cython components: Files like _native.pyx, _native.pxd, _native_shim.py, and _native.pyxbld indicate optimized, low-level performance parts.
- Additional modules: _statmoments_impl.py and _version.py, which likely manage the core calculations and versioning.
- Benchmarks and tests: The "benchmark" and "tests" directories show how performance and correctness are validated.
- The examples directory shows how both univariate (univar.py) and bivariate (bivar.py) tools can be used from scripts.

##### README for Additional Insights
- The README explains functionality, offering both a Python API and command-line interface.
- Detailed usage examples illustrate how inputs go through creation, updating of the engines, and finally extraction of statistical moments or performing t-tests.
- It also highlights performance considerations such as using dense matrix accumulators, handling streaming data, and integration with high-performance BLAS (or GPU-based) libraries.

a. Main Components to Include:
- Client Interface Layer:
- Command-Line Interface (CLI) tools (e.g., statmoments/univar.py and statmoments/bivar.py modules)
- Python API for direct usage in user applications
- Computation Engine:
- Univariate Processing Module (Univar class in univar.py)
- Bivariate Processing Module (Bivar class in bivar.py)
- Native/Cython Acceleration Module:
- Cython modules and native wrappers (_native.pyx, _native.pxd, etc.) that perform optimized calculation.
- Statistical Test Module:
- Module (stattests.py) which performs Welch’s t-tests and possibly other statistical validations.
- Data Input and Flow:
- Update methods taking streaming data batches and classifications.
- A pathway showing data flow from ingestion, through update, to moment computation.
- External Dependencies:
- Integration with BLAS libraries and optional GPU acceleration through nvmath-python.

b. Relationships and Interactions:
- Client (either via CLI or API) calling on the Univar/Bivar engine(s).
- Optionally, indicate that benchmark and test components separately validate both performance and correctness (perhaps as external verification blocks).

c. Highlighted Architectural Patterns or Design Principles:
- Single-pass streaming computation: Emphasize that data flows through the system in one pass for efficiency.
- Layered (or modular) design: Separation between the API/CLI layer, computation engine, and native optimization layer.
- High-performance optimization: Use of Cython modules and external BLAS libraries for numerical computation, and an emphasis on memory efficiency via dense matrix representations.

d. Relevant Technologies/Frameworks:
- Python as the primary language.
- Cython (files with .pyx, .pxd, etc.) for the native modules.
- BLAS libraries (including GPU-based options via nvmath-python) for optimized matrix operations.
- Standard CI/CD tools indicated by .github workflows, suggesting automated testing/integration practices.

##### Specific Elements
- Use clear labels for each component (e.g., "CLI Interface", "Univariate Engine", "Bivariate Engine", "Cython Acceleration Layer", "Statistical Test Module").

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client Interface"
        CLI["CLI/Python API Interface"] 
        EX1["Example: Univar Usage"]
        EX2["Example: Bivar Usage"]
    end

    subgraph "Computation Engine"
        UNI["Univariate Engine"]
        BIV["Bivariate Engine"]
    end

    subgraph "Native Acceleration Layer"
        NATIVE1["Cython Module (_native.pyx)"]
        NATIVE2["Cython Declaration (_native.pxd)"]
        NATIVE3["Build Configuration (_native.pyxbld)"]
        NATIVE4["Shim (_native_shim.py)"]
        NATIVE5["Implementation (_statmoments_impl.py)"]
    end

    STATT["Statistical Test Module"]
    BENCH["Benchmarking & Performance Testing"]
    VERSION["Version Management"]
    EXT["External BLAS & GPU (nvmath-python)"]

    %% Data Flow
    CLI --> UNI
    CLI --> BIV
    EX1 --> UNI
    EX2 --> BIV
    UNI -->|"compute"| NATIVE1
    UNI --> NATIVE2
    UNI --> NATIVE3
    UNI --> NATIVE4
    UNI --> NATIVE5
    BIV -->|"compute"| NATIVE1
    BIV --> NATIVE2
    BIV --> NATIVE3
    BIV --> NATIVE4
    BIV --> NATIVE5
    UNI -->|"statTest"| STATT
    BIV --> STATT
    BENCH --> UNI
    BENCH --> BIV
    VERSION -.-> UNI
    VERSION -.-> BIV
    EXT -.-> NATIVE1
    EXT -.-> NATIVE2
    EXT -.-> NATIVE3
    EXT -.-> NATIVE4
    EXT -.-> NATIVE5

    %% Styling
    classDef client fill:#f9d5e5,stroke:#b30059,stroke-width:2px;
    classDef engine fill:#d5e8d4,stroke:#82b366,stroke-width:2px;
    classDef native fill:#fff2cc,stroke:#f1c232,stroke-width:2px;
    classDef stat fill:#d0e0e3,stroke:#6d9eeb,stroke-width:2px;
    classDef bench fill:#e4dfda,stroke:#a64d79,stroke-width:2px;
    classDef version fill:#ead1dc,stroke:#ac4f8e,stroke-width:2px;
    classDef external fill:#d9e1f2,stroke:#6fa8dc,stroke-width:2px;

    class CLI,EX1,EX2 client;
    class UNI,BIV engine;
    class NATIVE1,NATIVE2,NATIVE3,NATIVE4,NATIVE5 native;
    class STATT stat;
    class BENCH bench;
    class VERSION version;
    class EXT external;

    %% Click Events
    click CLI "https://github.com/akochepasov/statmoments/blob/main/statmoments/univar.py"
    click UNI "https://github.com/akochepasov/statmoments/blob/main/statmoments/univar.py"
    click BIV "https://github.com/akochepasov/statmoments/blob/main/statmoments/bivar.py"
    click EX1 "https://github.com/akochepasov/statmoments/blob/main/examples/univar.py"
    click EX2 "https://github.com/akochepasov/statmoments/blob/main/examples/bivar.py"
    click NATIVE1 "https://github.com/akochepasov/statmoments/blob/main/statmoments/_native.pyx"
    click NATIVE2 "https://github.com/akochepasov/statmoments/blob/main/statmoments/_native.pxd"
    click NATIVE3 "https://github.com/akochepasov/statmoments/blob/main/statmoments/_native.pyxbld"
    click NATIVE4 "https://github.com/akochepasov/statmoments/blob/main/statmoments/_native_shim.py"
    click NATIVE5 "https://github.com/akochepasov/statmoments/blob/main/statmoments/_statmoments_impl.py"
    click STATT "https://github.com/akochepasov/statmoments/blob/main/statmoments/stattests.py"
    click BENCH "https://github.com/akochepasov/statmoments/tree/main/statmoments/benchmark"
    click VERSION "https://github.com/akochepasov/statmoments/blob/main/statmoments/_version.py"
```
