How dice-group/dice-embeddings works
Project type and purpose
• This is an open-source Python framework & CLI tool for hardware-agnostic large-scale knowledge graph embeddings.
• It provides:
• Data ingestion & preprocessing (pandas / polars / rdflib)
• Multiple embedding models (Clifford, ComplEx, TransE, etc.)
• Pluggable training backends (torchCPU, PyTorchLightning, DDP, model parallelism)
• Evaluation, query answering, link prediction, literal prediction
• Deployment via Huggingface Gradio & Qdrant vector DB
File structure
• Top-level: CLI entrypoint (dicee/__main__.py), Dockerfile, setup.py, tests, docs, examples
• Core package (dicee/):
• config.py, executer.py, callbacks.py, evaluator.py, analyse_experiments.py
• dataset & KG modules (dataset_classes.py, read_preprocess_save_load_kg/)
• model implementations (models/…)
• trainer implementations (trainer/torch_trainer.py, torch_trainer_ddp.py, model_parallelism.py)
• utilities: static_funcs, query_generator, knowledge_graph_embeddings, scripts/index_serve.py
• docs/ & examples/ illustrate usage, benchmarks, deployment
README for insights
• Data backends: pandas / modin / polars or rdflib for RDF
• ML backends: PyTorch + PyTorchLightning + optional FairScale/DeepSpeed
• Deployment: Huggingface Gradio, Qdrant (FastAPI/Uvicorn)
• Supports: CLI flags, Python API (dicee.executer.Execute), KGE class for inference & deployment
Main components & relationships
a. Components
• CLI & Config layer (dicee/__main__.py → config.py)
• Pipeline orchestrator (Execute in executer.py)
• Data Ingestion & Preprocessing (read_preprocess_save_load_kg/* + dataset_classes.py)
• KG abstraction (knowledge_graph.py)
• Model registry (models/* + abstracts.py)
• Training engine (trainer/* + callbacks.py)
• Evaluation (evaluator.py + eval_static_funcs.py)
• Query answering (query_generator.py + knowledge_graph_embeddings.py)
• Deployment & Serving (scripts/index_serve.py, KGE.deploy → Gradio, Qdrant client)
• External services: Qdrant vector DB, Huggingface Gradio, SPARQL endpoints
b. Data flow & interactions
1. User invokes dicee CLI or Python API → parsed into a Namespace (config).
2. Execute.start() reads triples via chosen backend → builds internal KG.
3. Executes training through selected Trainer → loads embedding model from models/*.
4. During training, callbacks handle periodic evaluation, SWA/ASWA, checkpointing.
5. On completion, produces embeddings, checkpoints, evaluation reports.
6. Optionally invoke KGE(path).answer_multi_hop_query() or predict_topk().
7. For deployment: KGE.deploy() spins up a Gradio interface; dicee_vector_db script indexes into Qdrant and launches FastAPI service.
c. Architectural patterns & principles
• Modular plugin architecture: add new reading backends, models, scoring techniques, trainers without touching core.
• Abstract base classes define common interfaces (abstracts.py).
• Callback pattern for training extensions (periodic eval, SWA, ASWA).
• Clear separation of concerns: ingestion vs. modeling vs. evaluation vs. deployment.
d. Technologies, frameworks & libraries
• Python packaging & CLI: setuptools, entry_points
• Data: pandas, modin/polars, rdflib
• ML: PyTorch, PyTorchLightning, FairScale, DeepSpeed
• Deployment: Huggingface Gradio, FastAPI, Uvicorn, Qdrant-client, Docker
• Documentation: Sphinx
Detail
• Sub-components (readers, preprocessors, trainer types, model categories).
• Plugin points: reading backend, scoring technique, trainer class, model choice.
• Callback chain during training (periodic eval → checkpoint → SWA/ASWA).