# drexalt/sparsecity architecture

> Architecture diagram of the GitHub repository drexalt/sparsecity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/drexalt/sparsecity
- Repository: https://github.com/drexalt/sparsecity

## Overview

##### Main Components
a. Configuration Layer:
- “conf” directory that contains multiple YAML configuration files for different models (e.g., cocondenser, distilbert, modernbert).
- This layer provides settings for training parameters, model architectures, and possibly profiling.
b. Data Handling:
- “dataset_transform.py” script and the “src/sparsecity/data” package (with files like “dataset.py”) that manage dataset preparation, transformation, and loading.
c. Models:
- “src/sparsecity/models” hierarchy with its two key parts:
– Base Models (“base_models”) such as “bert.py” and “flash_bert.py.”
– Specialized models (“splade_models”), including the “splade.py” and the “model_registry.py” which likely facilitates model instantiation.
d. Training:
- Training logic in “splade_train.py” and the training package in “src/sparsecity/training.” Note the presence of trainers (“trainer.py” and “sparse_trainer.py”) and loss definitions (“losses.py”), which together define the optimization and execution of training routines.
e. Evaluation:
- “evaluation” module (“st_wrapper.py” and “validate.py”) in “src/sparsecity/evaluation,” responsible for performance tests and validation of the trained models.
f. Ancillary Scripts & Environment Setup:
- Tools like “setup_env.sh” and “pyproject.toml” indicate dependency management and environment configuration.
- “kernel_test_2.py” may represent experiments or quick tests for verifying parts of the system.

##### Component Relationships and Interactions
- The configuration layer feeds parameters into the training and model modules. For example, YAML files from the “conf” directory configure model architectures and training hyperparameters.
- The data handling component provides preprocessed input which is passed to both the trainer and evaluation modules.
- The trainer modules (including loss functions) orchestrate the training process by interacting with the models (instantiated per configurations) and then possibly passing outputs or model snapshots to the evaluation system.
- The evaluation component uses both the trained models and the test datasets to derive performance metrics.

##### Architectural Patterns and Design Principles
- Modular design: each major concern (data, model, training, evaluation, configuration) is isolated in its own module.
- Separation of concerns and configuration-driven design: the YAML files provide a flexible means to switch models and training settings without modifying code.
- Extendability: the “model_registry.py” within the splade models indicates that new model types can be registered dynamically, suggesting an extensible plugin-like architecture.
- Emphasis on reproducibility: environment setup and structured configuration files support a reproducible and controlled training environment.

##### Relevant Technologies & Frameworks
- While not explicitly stated in the file tree, the presence of “pyproject.toml” hints at a modern Python environment possibly using tools like Poetry or similar dependency managers.
- Deep learning frameworks (most likely PyTorch or TensorFlow) are implicitly suggested by the model training and evaluation packages.
- YAML is used for configuration management, ensuring that parameter tuning does not require code changes.

##### Elements
- Clear labels for each major component:
– Configuration (YAML files in “conf”)
– Data Handling (dataset transformation scripts and “data” module)
– Model Layer split into Base Models and Specialized “splade_models”
– Training Module (trainer scripts and loss functions)
– Evaluation Module (validation and evaluation wrappers)
– From Configuration → Models & Training
– From Data Handling → Training and Evaluation

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration Layer"
        Config["Configuration (YAML)"]:::config
    end

    subgraph "Data Handling Layer"
        DataTransform["Data Transform (dataset_transform.py)"]:::data
        DataModule["Data Module (dataset.py)"]:::data
    end

    subgraph "Model Layer"
        BaseModels["Base Models (bert.py, flash_bert.py)"]:::model
        SpladeModels["Splade Models (splade.py, model_registry.py, memory_efficient.py)"]:::model
    end

    subgraph "Training Module"
        MainTrain["Main Training Script (splade_train.py)"]:::train
        TrainPack["Training Package (trainer.py, sparse_trainer.py, losses.py)"]:::train
    end

    subgraph "Evaluation Module"
        Eval["Evaluation (st_wrapper.py, validate.py)"]:::eval
    end

    subgraph "Ancillary Setup"
        SetupEnv["setup_env.sh"]:::ancillary
        PyProject["pyproject.toml"]:::ancillary
        KernelTest["kernel_test_2.py"]:::ancillary
    end

    %% Component Relationships
    Config -->|"configures"| BaseModels
    Config -->|"configures"| SpladeModels
    Config -->|"configures"| TrainPack

    DataTransform -->|"provides_data"| TrainPack
    DataModule -->|"provides_data"| TrainPack
    DataModule -->|"supplies_data"| Eval

    BaseModels -->|"used_by"| TrainPack
    SpladeModels -->|"used_by"| TrainPack
    MainTrain -->|"initiates"| TrainPack

    TrainPack -->|"generates_model"| Eval

    %% Ancillary connections (supporting environment)
    SetupEnv --- Config
    PyProject --- Config
    KernelTest --- MainTrain

    %% Click Events
    click Config "https://github.com/drexalt/sparsecity/tree/master/conf/"
    click DataTransform "https://github.com/drexalt/sparsecity/blob/master/dataset_transform.py"
    click DataModule "https://github.com/drexalt/sparsecity/blob/master/src/sparsecity/data/dataset.py"
    click BaseModels "https://github.com/drexalt/sparsecity/tree/master/src/sparsecity/models/base_models/"
    click SpladeModels "https://github.com/drexalt/sparsecity/tree/master/src/sparsecity/models/splade_models/"
    click MainTrain "https://github.com/drexalt/sparsecity/blob/master/splade_train.py"
    click TrainPack "https://github.com/drexalt/sparsecity/tree/master/src/sparsecity/training/"
    click Eval "https://github.com/drexalt/sparsecity/tree/master/src/sparsecity/evaluation/"
    click SetupEnv "https://github.com/drexalt/sparsecity/blob/master/setup_env.sh"
    click PyProject "https://github.com/drexalt/sparsecity/blob/master/pyproject.toml"
    click KernelTest "https://github.com/drexalt/sparsecity/blob/master/kernel_test_2.py"

    %% Styles
    classDef config fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef data fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef model fill:#ffe6cc,stroke:#cc7a00,stroke-width:2px;
    classDef train fill:#d4edda,stroke:#28a745,stroke-width:2px;
    classDef eval fill:#e2d6f3,stroke:#6f42c1,stroke-width:2px;
    classDef ancillary fill:#e6e6e6,stroke:#808080,stroke-width:2px;
```
