How wangyuxinwhy/uniem works
Main Components
a. Core Components:
• Model Training System
• Fine-tuning System
• Evaluation System (MTEB-zh)
• Integration Layer
b. Key Modules:
• Data Processing Pipeline
• Model Architecture
• Training Infrastructure
• Evaluation Framework
Top Level (User Interaction):
• API Layer (Public Interfaces)
• FineTuner API
• SentenceTransformer Integration
Middle Level (Core Processing):
• Model Management
• Model Training
• Fine-tuning
• Model Evaluation
Bottom Level (Infrastructure):
• Data Processing
• Dataset Handlers
• Utilities & Support
Component Relationships
• Blue: Core model components
• Green: Data processing components
• Orange: API and integration layers
• Gray: Utility components
Specific Implementation Details
a. Model Pipeline:
• Flow from raw data → processing → training → evaluation
• M3E model variants (small, base, large)
• Demonstrate the integration with sentence-transformers
b. Fine-tuning System:
• FineTuner component and its interaction with various model types
• Support for different model architectures (sentence_transformers, text2vec, SGPT)
c. Evaluation Framework:
• MTEB-zh evaluation system
• Connection to different evaluation tasks (classification, retrieval)
Key Technical Elements
• HuggingFace Integration
• Python Package Structure
• Testing Framework
• CI/CD Pipeline
Implementation Notes
Use standardized symbols:
• Rectangles: Main components
• Diamonds: Decision points