# wangyuxinwhy/uniem architecture

> Architecture diagram of the GitHub repository wangyuxinwhy/uniem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wangyuxinwhy/uniem
- Repository: https://github.com/wangyuxinwhy/uniem

## Overview

##### Main Components

a. Core Components:
- Model Training System
- Fine-tuning System
- Evaluation System (MTEB-zh)
- Integration Layer

b. Key Modules:
- Data Processing Pipeline
- Model Architecture
- Training Infrastructure
- Evaluation Framework

**Top Level (User Interaction):**

- API Layer (Public Interfaces)
- FineTuner API
- SentenceTransformer Integration

**Middle Level (Core Processing):**

- Model Management
- Model Training
- Fine-tuning
- Model Evaluation

**Bottom Level (Infrastructure):**

- Data Processing
- Dataset Handlers

- Utilities & Support

##### Component Relationships

- Blue: Core model components
- Green: Data processing components
- Orange: API and integration layers
- Gray: Utility components

##### Specific Implementation Details

a. Model Pipeline:
- Flow from raw data → processing → training → evaluation
- M3E model variants (small, base, large)
- Demonstrate the integration with sentence-transformers

b. Fine-tuning System:
- FineTuner component and its interaction with various model types
- Support for different model architectures (sentence_transformers, text2vec, SGPT)

c. Evaluation Framework:
- MTEB-zh evaluation system
- Connection to different evaluation tasks (classification, retrieval)

##### Key Technical Elements

- HuggingFace Integration
- Python Package Structure
- Testing Framework
- CI/CD Pipeline

##### Implementation Notes

Use standardized symbols:
- Rectangles: Main components
- Diamonds: Decision points

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and classes
    classDef api fill:#FFA07A,stroke:#333,stroke-width:2px
    classDef core fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef data fill:#98FB98,stroke:#333,stroke-width:2px
    classDef infra fill:#DDD,stroke:#333,stroke-width:2px
    classDef eval fill:#DDA0DD,stroke:#333,stroke-width:2px

    %% Top Level - API Layer
    subgraph API
        FT["FineTuner API"]:::api
        ST["SentenceTransformer Integration"]:::api
    end

    %% Middle Level - Core Processing
    subgraph Core
        MT["Model Training System"]:::core
        FTS["Fine-tuning System"]:::core
        MA["Model Architecture"]:::core
        
        subgraph Evaluation
            MTEB["MTEB-zh Evaluation"]:::eval
            ET["Evaluation Tasks"]:::eval
        end
    end

    %% Bottom Level - Infrastructure
    subgraph Infrastructure
        DP["Data Processing Pipeline"]:::data
        DH["Dataset Handlers"]:::data
        TF["Testing Framework"]:::infra
        CI["CI/CD Pipeline"]:::infra
    end

    %% Relationships
    DP --> DH
    DH --> MT
    MT --> MA
    MA --> FTS
    FTS --> MTEB
    MTEB --> ET
    FT --> FTS
    ST --> MA
    MA --> TF
    FTS --> TF
    CI --> TF

    %% Click events
    click MT "https://github.com/wangyuxinwhy/uniem/blob/main/uniem/trainer.py"
    click FTS "https://github.com/wangyuxinwhy/uniem/blob/main/uniem/finetuner.py"
    click MTEB "https://github.com/wangyuxinwhy/uniem/blob/main/mteb-zh/mteb_zh/tasks.py"
    click ST "https://github.com/wangyuxinwhy/uniem/blob/main/uniem/integration/sentence_transformers_wrapper.py"
    click DP "https://github.com/wangyuxinwhy/uniem/blob/main/uniem/data.py"
    click MA "https://github.com/wangyuxinwhy/uniem/blob/main/uniem/model.py"
    click TF "https://github.com/wangyuxinwhy/uniem/tree/main/tests"
    click CI "https://github.com/wangyuxinwhy/uniem/tree/main/.github/workflows"

    %% Legend
    subgraph Legend
        L1["API Components"]:::api
        L2["Core Components"]:::core
        L3["Data Components"]:::data
        L4["Infrastructure"]:::infra
        L5["Evaluation"]:::eval
    end
```
