# dice-group/dice-embeddings architecture

> Architecture diagram of the GitHub repository dice-group/dice-embeddings, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dice-group/dice-embeddings
- Repository: https://github.com/dice-group/dice-embeddings

## Overview

##### Project type and purpose
- This is an open-source Python framework & CLI tool for hardware-agnostic large-scale knowledge graph embeddings.
- It provides:
- Data ingestion & preprocessing (pandas / polars / rdflib)
- Multiple embedding models (Clifford, ComplEx, TransE, etc.)
- Pluggable training backends (torchCPU, PyTorchLightning, DDP, model parallelism)
- Evaluation, query answering, link prediction, literal prediction
- Deployment via Huggingface Gradio & Qdrant vector DB

##### File structure
- Top-level: CLI entrypoint (`dicee/__main__.py`), Dockerfile, setup.py, tests, docs, examples
- Core package (`dicee/`):
- config.py, executer.py, callbacks.py, evaluator.py, analyse_experiments.py
- dataset & KG modules (`dataset_classes.py`, `read_preprocess_save_load_kg/`)
- model implementations (`models/…`)
- trainer implementations (`trainer/torch_trainer.py`, `torch_trainer_ddp.py`, `model_parallelism.py`)
- utilities: static_funcs, query_generator, knowledge_graph_embeddings, scripts/index_serve.py
- docs/ & examples/ illustrate usage, benchmarks, deployment

##### README for insights
- Data backends: pandas / modin / polars or rdflib for RDF
- ML backends: PyTorch + PyTorchLightning + optional FairScale/DeepSpeed
- Deployment: Huggingface Gradio, Qdrant (FastAPI/Uvicorn)
- Supports: CLI flags, Python API (`dicee.executer.Execute`), KGE class for inference & deployment

##### Main components & relationships
a. Components
- CLI & Config layer (dicee/__main__.py → config.py)
- Pipeline orchestrator (Execute in executer.py)
- Data Ingestion & Preprocessing (read_preprocess_save_load_kg/* + dataset_classes.py)
- KG abstraction (knowledge_graph.py)
- Model registry (models/* + abstracts.py)
- Training engine (trainer/* + callbacks.py)
- Evaluation (evaluator.py + eval_static_funcs.py)
- Query answering (query_generator.py + knowledge_graph_embeddings.py)
- Deployment & Serving (scripts/index_serve.py, KGE.deploy → Gradio, Qdrant client)
- External services: Qdrant vector DB, Huggingface Gradio, SPARQL endpoints

b. Data flow & interactions
1. User invokes `dicee` CLI or Python API → parsed into a Namespace (config).
2. `Execute.start()` reads triples via chosen backend → builds internal KG.
3. Executes training through selected Trainer → loads embedding model from models/*.
4. During training, callbacks handle periodic evaluation, SWA/ASWA, checkpointing.
5. On completion, produces embeddings, checkpoints, evaluation reports.
6. Optionally invoke `KGE(path).answer_multi_hop_query()` or `predict_topk()`.
7. For deployment: `KGE.deploy()` spins up a Gradio interface; `dicee_vector_db` script indexes into Qdrant and launches FastAPI service.

c. Architectural patterns & principles
- Modular plugin architecture: add new reading backends, models, scoring techniques, trainers without touching core.
- Abstract base classes define common interfaces (abstracts.py).
- Callback pattern for training extensions (periodic eval, SWA, ASWA).
- Clear separation of concerns: ingestion vs. modeling vs. evaluation vs. deployment.

d. Technologies, frameworks & libraries
- Python packaging & CLI: setuptools, entry_points
- Data: pandas, modin/polars, rdflib
- ML: PyTorch, PyTorchLightning, FairScale, DeepSpeed
- Deployment: Huggingface Gradio, FastAPI, Uvicorn, Qdrant-client, Docker
- Documentation: Sphinx

##### Detail
- Sub-components (readers, preprocessors, trainer types, model categories).
- Plugin points: reading backend, scoring technique, trainer class, model choice.
- Callback chain during training (periodic eval → checkpoint → SWA/ASWA).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Ingestion Layer
    subgraph "Data & Ingestion Layer"
        direction TB
        CLI["CLI (__main__.py)"]:::data
        Config["Config (config.py)"]:::data
        Executor["Pipeline Orchestrator (executer.py)"]:::data
        subgraph "Data Ingestion & Preprocessing"
            direction TB
            DataClasses["dataset_classes.py"]:::data
            Read["read_from_disk.py"]:::data
            Preprocess["preprocess.py"]:::data
            SaveLoad["save_load_disk.py"]:::data
            Util["util.py"]:::data
        end
        KG["Knowledge Graph Abstraction (knowledge_graph.py)"]:::data
    end

    %% Data Stores
    DataTriples["Input Triples"]:::datastore
    InternalKG["Internal KG (in-memory)"]:::datastore

    %% Training & Model Layer
    subgraph "Training & Model Layer"
        direction TB
        subgraph "Model Registry & Implementations"
            direction TB
            Abstracts["abstracts.py"]:::training
            BaseModel["base_model.py"]:::training
            Clifford["clifford.py"]:::training
            Complex["complex.py"]:::training
            DualE["dualE.py"]:::training
            Ensemble["ensemble.py"]:::training
            FunctionSpace["function_space.py"]:::training
            Literal["literal.py"]:::training
            Octonion["octonion.py"]:::training
            Pykeen["pykeen_models.py"]:::training
            Quaternion["quaternion.py"]:::training
            Real["real.py"]:::training
            Transformers["transformers.py"]:::training
            Adopt["adopt.py"]:::training
            StaticFuncsModel["static_funcs.py"]:::training
        end
        subgraph "Training Engine & Callbacks"
            direction TB
            DiceTrainer["dice_trainer.py"]:::training
            TorchTrainer["torch_trainer.py"]:::training
            DDPtrainer["torch_trainer_ddp.py"]:::training
            ModelParallel["model_parallelism.py"]:::training
            Callbacks["callbacks.py"]:::training
        end
    end

    %% Training Outputs
    Checkpoints["Model Checkpoints"]:::datastore
    Embeddings["Embeddings (tensor outputs)"]:::datastore

    %% Evaluation & Query Layer
    subgraph "Evaluation & Query Layer"
        direction TB
        Eval["evaluator.py & eval_static_funcs.py"]:::eval
        Query["query_generator.py & knowledge_graph_embeddings.py"]:::eval
    end

    %% Deployment & Serving Layer
    subgraph "Deployment & Serving Layer"
        direction TB
        Deploy["Deployment & Serving (index_serve.py & run.py)"]:::deploy
    end

    %% External Services
    SPARQL["SPARQL Endpoint"]:::external
    QdrantServer["Qdrant Vector DB"]:::external
    Gradio["Huggingface Gradio"]:::external

    %% Flows
    CLI -->|"flags/API call"| Config
    Config -->|"config Namespace"| Executor
    Executor -->|"reads triples"| DataTriples
    DataTriples -->|"uses backends"| DataClasses
    DataClasses --> Read
    DataClasses --> Preprocess
    DataClasses --> SaveLoad
    DataClasses --> Util
    Preprocess --> KG
    Read --> KG
    SaveLoad --> KG
    Util --> KG
    KG -->|"builds KG object"| InternalKG
    InternalKG -->|"provides graph"| DiceTrainer
    KG -->|"; graph API"| TorchTrainer
    Abstracts --> BaseModel
    BaseModel --> Clifford
    BaseModel --> Complex
    BaseModel --> DualE
    BaseModel --> Ensemble
    BaseModel --> FunctionSpace
    BaseModel --> Literal
    BaseModel --> Octonion
    BaseModel --> Pykeen
    BaseModel --> Quaternion
    BaseModel --> Real
    BaseModel --> Transformers
    BaseModel --> Adopt
    BaseModel --> StaticFuncsModel
    DiceTrainer --> Callbacks
    TorchTrainer --> Callbacks
    DDPtrainer --> Callbacks
    ModelParallel --> Callbacks
    DiceTrainer -->|trains with| BaseModel
    DiceTrainer -->|outputs tensors| Embeddings
    TorchTrainer -->|outputs tensors| Embeddings
    DDPtrainer -->|outputs tensors| Embeddings
    ModelParallel -->|outputs tensors| Embeddings
    Embeddings -->|"save to"| Checkpoints
    Checkpoints -->|"load for eval"| Eval
    Embeddings -->|"vectors for query"| Query
    Eval -->|"metrics"| Executor
    Query -->|"answers"| Executor
    Query -->|"index vectors"| Deploy
    Deploy -->|"serves via"| Gradio
    Deploy -->|"indexes via"| QdrantServer

    %% External Calls
    KG -->|"SPARQL queries"| SPARQL
    Deploy -->|"uses API"| QdrantServer

    %% Click Events
    click CLI "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/__main__.py"
    click Config "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/config.py"
    click Executor "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/executer.py"
    click DataClasses "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/dataset_classes.py"
    click Read "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/read_preprocess_save_load_kg/read_from_disk.py"
    click Preprocess "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/read_preprocess_save_load_kg/preprocess.py"
    click SaveLoad "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/read_preprocess_save_load_kg/save_load_disk.py"
    click Util "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/read_preprocess_save_load_kg/util.py"
    click Abstracts "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/abstracts.py"
    click BaseModel "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/base_model.py"
    click Clifford "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/clifford.py"
    click Complex "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/complex.py"
    click DualE "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/dualE.py"
    click Ensemble "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/ensemble.py"
    click FunctionSpace "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/function_space.py"
    click Literal "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/literal.py"
    click Octonion "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/octonion.py"
    click Pykeen "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/pykeen_models.py"
    click Quaternion "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/quaternion.py"
    click Real "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/real.py"
    click Transformers "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/transformers.py"
    click Adopt "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/adopt.py"
    click StaticFuncsModel "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/models/static_funcs.py"
    click DiceTrainer "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/trainer/dice_trainer.py"
    click TorchTrainer "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/trainer/torch_trainer.py"
    click DDPtrainer "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/trainer/torch_trainer_ddp.py"
    click ModelParallel "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/trainer/model_parallelism.py"
    click Callbacks "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/callbacks.py"
    click Eval "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/evaluator.py"
    click Query "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/query_generator.py"
    click Query "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/knowledge_graph_embeddings.py"
    click Deploy "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/scripts/index_serve.py"
    click Deploy "https://github.com/dice-group/dice-embeddings/blob/develop/dicee/scripts/run.py"

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef training fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef eval fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef deploy fill:#e1bee7,stroke:#8e24aa,color:#4a148c
    classDef datastore fill:#b2dfdb,stroke:#00796b,color:#004d40
    classDef external fill:#f0f4c3,stroke:#c0ca33,color:#827717
```
