# maelic/relateanything architecture

> Architecture diagram of the GitHub repository maelic/relateanything, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maelic/relateanything
- Repository: https://github.com/maelic/relateanything

## Overview

RelateAnything predicts scored subject–predicate–object relations from an image and supplied boxes or masks, with caller-selected predicate vocabularies and optional spatial/semantic graph decomposition. Its principal runtime path is the Python API into the relation model, followed by score handling and ranked triplets. Separate branches cover detector-assisted deployment, dataset and annotation preparation, model training, and evaluation. The README documents additional deployment targets; only the Python deployment pipeline is represented in detail here because the sampled deployment source is partial.

Main entry points: [`relsgg/model/relsgg.py`](https://github.com/maelic/relateanything/blob/main/relsgg/model/relsgg.py) (RelSGG model), [`relsgg/api.py`](https://github.com/maelic/relateanything/blob/main/relsgg/api.py) (Python API), [`deploy/pipeline.py`](https://github.com/maelic/relateanything/blob/main/deploy/pipeline.py) (Scene pipeline), [`train.py`](https://github.com/maelic/relateanything/blob/main/train.py) (Training entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference API"]
  node_api["Python API<br/>[api.py]"]
  node_text["Predicate embeddings<br/>[student.py]"]
  node_triplets["Ranked triplets<br/>[api.py]"]
end

subgraph group_model["Relation model"]
  node_model_core["RelSGG model<br/>[relsgg.py]"]
  node_backbone["Image backbone<br/>[backbone.py]"]
  node_pool["Region pooling<br/>[pooling.py]"]
  node_sampler["Pair sampler<br/>[sampler.py]"]
  node_geometry["Pair geometry<br/>[geometry.py]"]
  node_transformer["Scene transformer<br/>[transformer.py]"]
  node_vocab_head["Vocabulary head<br/>[vocab_head.py]"]
  node_score["Score contract<br/>[scoring.py]"]
end

subgraph group_learning["Data and training"]
  node_source_data[("Source datasets")]
  node_pack[("Training packs")]
  node_dataset["Dataset loader<br/>[dataset.py]"]
  node_train_setup["Training setup<br/>[setup.py]"]
  node_train_entry["Training entry<br/>[train.py]"]
  node_engine["Training engine<br/>[engine.py]"]
  node_checkpoint["Model checkpoint<br/>[checkpoint.py]"]
  node_annotation["Relation annotation"]
end

subgraph group_deployment["Deployment"]
  node_pipeline["Scene pipeline<br/>[pipeline.py]"]
  node_detector["Object detector"]
  node_runtime["Exported runtimes<br/>[runtime.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluator["Relation evaluator<br/>[evaluator.py]"]
  node_eval_data[("Evaluation suites")]
end

node_user(("Image and regions"))
node_train_user(("Training operator"))
node_deploy_user(("Deployment user"))
node_eval_user(("Evaluation operator"))

node_user -->|"provides image and regions"| node_api
node_api -->|"runs prediction"| node_model_core
node_api -->|"encodes predicates"| node_text
node_api -->|"calibrates scores"| node_score
node_model_core -->|"extracts image features"| node_backbone
node_model_core -->|"pools region features"| node_pool
node_model_core -->|"selects pairs"| node_sampler
node_model_core -->|"encodes pair geometry"| node_geometry
node_model_core -->|"refines pair context"| node_transformer
node_model_core -->|"scores predicates"| node_vocab_head
node_text -->|"supplies embeddings"| node_vocab_head
node_api -->|"returns ranked relations"| node_triplets
node_train_user -->|"starts training"| node_train_entry
node_source_data -->|"converted and packed"| node_pack
node_annotation -->|"generates annotations"| node_pack
node_pack -->|"loads samples"| node_dataset
node_train_entry -->|"configures training"| node_train_setup
node_train_entry -->|"runs training loop"| node_engine
node_train_setup -->|"builds model"| node_model_core
node_engine -->|"consumes batches"| node_dataset
node_engine -->|"saves weights"| node_checkpoint
node_deploy_user -->|"submits frames"| node_pipeline
node_pipeline -->|"detects regions"| node_detector
node_pipeline -->|"predicts relations"| node_api
node_pipeline -.->|"uses backend"| node_runtime
node_eval_user -->|"runs evaluation"| node_evaluator
node_eval_data -->|"provides benchmarks"| node_evaluator
node_evaluator -->|"evaluates predictions"| node_model_core

click node_api "https://github.com/maelic/relateanything/blob/main/relsgg/api.py"
click node_text "https://github.com/maelic/relateanything/blob/main/relsgg/text/student.py"
click node_model_core "https://github.com/maelic/relateanything/blob/main/relsgg/model/relsgg.py"
click node_backbone "https://github.com/maelic/relateanything/blob/main/relsgg/model/backbone.py"
click node_pool "https://github.com/maelic/relateanything/blob/main/relsgg/model/pooling.py"
click node_sampler "https://github.com/maelic/relateanything/blob/main/relsgg/model/sampler.py"
click node_geometry "https://github.com/maelic/relateanything/blob/main/relsgg/model/geometry.py"
click node_transformer "https://github.com/maelic/relateanything/blob/main/relsgg/model/transformer.py"
click node_vocab_head "https://github.com/maelic/relateanything/blob/main/relsgg/model/vocab_head.py"
click node_score "https://github.com/maelic/relateanything/blob/main/relsgg/scoring.py"
click node_triplets "https://github.com/maelic/relateanything/blob/main/relsgg/api.py"
click node_source_data "https://github.com/maelic/relateanything/tree/main/training"
click node_pack "https://github.com/maelic/relateanything/tree/main/relsgg/data"
click node_dataset "https://github.com/maelic/relateanything/blob/main/relsgg/data/dataset.py"
click node_train_setup "https://github.com/maelic/relateanything/blob/main/relsgg/training/setup.py"
click node_train_entry "https://github.com/maelic/relateanything/blob/main/train.py"
click node_engine "https://github.com/maelic/relateanything/blob/main/relsgg/training/engine.py"
click node_checkpoint "https://github.com/maelic/relateanything/blob/main/relsgg/checkpoint.py"
click node_annotation "https://github.com/maelic/relateanything/blob/main/datagen/sgg_vllm_generate.py"
click node_pipeline "https://github.com/maelic/relateanything/blob/main/deploy/pipeline.py"
click node_runtime "https://github.com/maelic/relateanything/blob/main/deploy/runtime.py"
click node_evaluator "https://github.com/maelic/relateanything/blob/main/relsgg/eval/evaluator.py"
click node_eval_data "https://github.com/maelic/relateanything/tree/main/relsgg/eval"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_text,node_triplets,node_deploy_user toneBlue
class node_model_core,node_backbone,node_pool,node_sampler,node_geometry,node_transformer,node_vocab_head,node_score toneAmber
class node_source_data,node_pack,node_dataset,node_train_setup,node_train_entry,node_engine,node_checkpoint,node_annotation toneMint
class node_pipeline,node_detector,node_runtime toneRose
class node_evaluator,node_eval_data,node_user,node_train_user,node_eval_user toneIndigo
```

## Components

### Inference API

- [Python API](https://github.com/maelic/relateanything/blob/main/relsgg/api.py): `relsgg/api.py`
- [Predicate embeddings](https://github.com/maelic/relateanything/blob/main/relsgg/text/student.py): `relsgg/text/student.py`
- [Ranked triplets](https://github.com/maelic/relateanything/blob/main/relsgg/api.py): `relsgg/api.py`

### Relation model

- [RelSGG model](https://github.com/maelic/relateanything/blob/main/relsgg/model/relsgg.py): `relsgg/model/relsgg.py`
- [Image backbone](https://github.com/maelic/relateanything/blob/main/relsgg/model/backbone.py): `relsgg/model/backbone.py`
- [Region pooling](https://github.com/maelic/relateanything/blob/main/relsgg/model/pooling.py): `relsgg/model/pooling.py`
- [Pair sampler](https://github.com/maelic/relateanything/blob/main/relsgg/model/sampler.py): `relsgg/model/sampler.py`
- [Pair geometry](https://github.com/maelic/relateanything/blob/main/relsgg/model/geometry.py): `relsgg/model/geometry.py`
- [Scene transformer](https://github.com/maelic/relateanything/blob/main/relsgg/model/transformer.py): `relsgg/model/transformer.py`
- [Vocabulary head](https://github.com/maelic/relateanything/blob/main/relsgg/model/vocab_head.py): `relsgg/model/vocab_head.py`
- [Score contract](https://github.com/maelic/relateanything/blob/main/relsgg/scoring.py): `relsgg/scoring.py`

### Data and training

- [Source datasets](https://github.com/maelic/relateanything/tree/main/training): `training`
- [Training packs](https://github.com/maelic/relateanything/tree/main/relsgg/data): `relsgg/data`
- [Dataset loader](https://github.com/maelic/relateanything/blob/main/relsgg/data/dataset.py): `relsgg/data/dataset.py`
- [Training setup](https://github.com/maelic/relateanything/blob/main/relsgg/training/setup.py): `relsgg/training/setup.py`
- [Training entry](https://github.com/maelic/relateanything/blob/main/train.py): `train.py`
- [Training engine](https://github.com/maelic/relateanything/blob/main/relsgg/training/engine.py): `relsgg/training/engine.py`
- [Model checkpoint](https://github.com/maelic/relateanything/blob/main/relsgg/checkpoint.py): `relsgg/checkpoint.py`
- [Relation annotation](https://github.com/maelic/relateanything/blob/main/datagen/sgg_vllm_generate.py): `datagen/sgg_vllm_generate.py`

### Deployment

- [Scene pipeline](https://github.com/maelic/relateanything/blob/main/deploy/pipeline.py): `deploy/pipeline.py`
- **Object detector**
- [Exported runtimes](https://github.com/maelic/relateanything/blob/main/deploy/runtime.py): `deploy/runtime.py`

### Evaluation

- [Relation evaluator](https://github.com/maelic/relateanything/blob/main/relsgg/eval/evaluator.py): `relsgg/eval/evaluator.py`
- [Evaluation suites](https://github.com/maelic/relateanything/tree/main/relsgg/eval): `relsgg/eval`

### Other components

- **Image and regions**
- **Training operator**
- **Deployment user**
- **Evaluation operator**

## Connections

- Image and regions → Python API: provides image and regions
- Python API → RelSGG model: runs prediction
- Python API → Predicate embeddings: encodes predicates
- Python API → Score contract: calibrates scores
- RelSGG model → Image backbone: extracts image features
- RelSGG model → Region pooling: pools region features
- RelSGG model → Pair sampler: selects pairs
- RelSGG model → Pair geometry: encodes pair geometry
- RelSGG model → Scene transformer: refines pair context
- RelSGG model → Vocabulary head: scores predicates
- Predicate embeddings → Vocabulary head: supplies embeddings
- Python API → Ranked triplets: returns ranked relations
- Training operator → Training entry: starts training
- Source datasets → Training packs: converted and packed
- Relation annotation → Training packs: generates annotations
- Training packs → Dataset loader: loads samples
- Training entry → Training setup: configures training
- Training entry → Training engine: runs training loop
- Training setup → RelSGG model: builds model
- Training engine → Dataset loader: consumes batches
- Training engine → Model checkpoint: saves weights
- Deployment user → Scene pipeline: submits frames
- Scene pipeline → Object detector: detects regions
- Scene pipeline → Python API: predicts relations
- Scene pipeline → Exported runtimes: uses backend
- Evaluation operator → Relation evaluator: runs evaluation
- Evaluation suites → Relation evaluator: provides benchmarks
- Relation evaluator → RelSGG model: evaluates predictions