# jfkback/hypencoder-paper architecture

> Architecture diagram of the GitHub repository jfkback/hypencoder-paper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jfkback/hypencoder-paper
- Repository: https://github.com/jfkback/hypencoder-paper

## Overview

##### Project Type and Purpose
- Hypencoder is an open-source machine‐learning library/toolkit for Information Retrieval (IR), implementing hypernetwork-based encoders (“q‐nets”) and bi‐encoder retrieval pipelines. It provides:
- Model definitions (Hypencoder, Q‐nets, similarity & loss functions)
- Training workflows (data loading, config management, training loops)
- Inference workflows (encoding documents, retrieving & evaluating)

##### File‐Structure Analysis
- Top‐Level:
- setup.py, LICENSE, README.md – packaging & documentation
- replication_commands.md – experiment orchestration
- imgs/ – architecture illustration
- Core Package “hypencoder_cb/”:
- modeling/ – hypernetwork models & loss definitions
- train/ – training entrypoint, argument parsing, configs, data collator
- inference/ – CLI scripts for encode, retrieve; neighbor_graph & retrieval logic
- utils/ – shared utilities (data I/O, tokenization, evaluation, torch helpers)
- Configuration & Artifacts:
- train/configs/*.yaml – experiment configurations (OmegaConf)
- JSONL, IR‐Datasets & external artifact storage (Google Drive & Huggingface Hub)

- External Actors & Services:
- User/Experimenter (CLI invocations)
- PyTorch & Transformers libraries
- IR-Datasets (dataset loader)
- Huggingface Hub & Google Drive (model & artifact storage)
- Core System Modules:
- CLI Layer (“encode.py”, “retrieve.py”, “train.py”)
- Inference Subsystem:
– Encoder Component (TextEncoder & HypencoderDualEncoder)
– Approximate & Exact Retrieval (neighbor_graph, retrieve.py)
– Evaluation Module (eval_utils, IR-Measures)
- Training Subsystem:
– Config Manager (OmegaConf parsing via args.py)
– Data Loader & Collator (datasets library + data_collator.py)
– Trainer (train.py orchestrating forward/backward, logging)
– Model Exporter (saving checkpoints to local or HF)
- Modeling Library:
– Hypencoder hypernetworks (hypencoder.py, q_net.py)
– Shared layers & loss functions (similarity_and_losses.py)
- Utility Layer:
- I/O utilities (jsonl_utils, io_utils)
- Tokenizer utilities & iteration helpers
- Torch helpers (device placement, seed setting)

##### Component Interactions & Data Flow
- Training Flow:
1. User runs `train.py` with a YAML config → args.py parses args & loads OmegaConf config
2. DataLoader (IR-Datasets or JSONL) fetches queries/passages → data_collator batches
3. Trainer loops: model (hypencoder) computes embeddings & losses → optimizer updates weights
4. Checkpoints & logs are written to disk and optionally pushed to Huggingface Hub
- Inference Flow (Encode):
1. User runs `encode.py` → instantiate HypencoderDualEncoder.from_pretrained(model_name)
2. Tokenizer processes text → TextEncoder and query_encoder produce embeddings
3. Embeddings saved as JSONL or binary files in `ENCODING_PATH`
- Inference Flow (Retrieve):
1. User runs `retrieve.py` with encoded items and query file
2. Query embeddings computed, neighbor_graph builds/indexes items (approximate search)
3. Q-nets score candidate passages → results written and optionally evaluated (via ir_measures)

##### Architectural Patterns & Principles
- Modular package structure separating modeling, training, inference, and utils
- CLI‐driven workflows for reproducibility & scriptability
- Configuration‐as‐code via OmegaConf YAML files
- Extensibility: custom q‐net converters plug into modeling/shared design
- Data abstraction: IR-Datasets & JSONL interchange formats

- Clear component names matching directory names
- CLI Entry Points
- Core Modules
- External Libraries/Services

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    User["User/Experimenter (CLI)"]:::external
    PyTorch["PyTorch & Transformers"]:::external
    IRDatasets["IR-Datasets"]:::external
    HFHub["Huggingface Hub & Google Drive"]:::external

    %% CLI Layer
    subgraph "CLI Entry Points"
        direction TB
        TrainCLI["train.py"]:::workflows
        EncodeCLI["encode.py"]:::workflows
        RetrieveCLI["retrieve.py"]:::workflows
    end

    %% Training Subsystem
    subgraph "Training Subsystem"
        direction TB
        Args["args.py"]:::workflows
        Configs["*.yaml"]:::workflows
        DataCollator["data_collator.py"]:::workflows
        Trainer["train.py"]:::workflows
    end

    %% Inference Subsystem
    subgraph "Inference Subsystem"
        direction TB
        ApproxRetrieve["approx_retrieve.py"]:::workflows
        NeighborGraph["neighbor_graph.py"]:::workflows
        SharedInf["shared.py"]:::workflows
    end

    %% Modeling Library
    subgraph "Modeling Library"
        direction TB
        HypencoderModel["hypencoder.py"]:::modeling
        QNet["q_net.py"]:::modeling
        ModelShared["shared.py"]:::modeling
        Losses["similarity_and_losses.py"]:::modeling
    end

    %% Utility Layer
    subgraph "Utility Layer"
        direction TB
        IOUtils["io_utils.py"]:::utils
        JSONLUtils["jsonl_utils.py"]:::utils
        EvalUtils["eval_utils.py"]:::utils
        TokenizerUtils["tokenizer_utils.py"]:::utils
        IteratorUtils["iterator_utils.py"]:::utils
        TorchUtils["torch_utils.py"]:::utils
        DataUtils["data_utils.py"]:::utils
    end

    %% Top-Level Files
    subgraph "Top-Level Project Files"
        direction TB
        Setup["setup.py"]:::external
        README["README.md"]:::external
        LICENSE["LICENSE"]:::external
        Replication["replication_commands.md"]:::external
        IMGS["imgs/"]:::external
    end

    %% Connections: User to CLI
    User -->|"runs"| TrainCLI
    User -->|"runs"| EncodeCLI
    User -->|"runs"| RetrieveCLI

    %% Training Flow
    TrainCLI -->|"parse args"| Args
    Args -->|"load config"| Configs
    Configs -->|"provide parameters"| Trainer
    Trainer -->|"uses collator"| DataCollator
    DataCollator -->|"batch data from"| IRDatasets
    Trainer -->|"calls model"| HypencoderModel
    HypencoderModel -->|"utilizes"| QNet
    QNet -->|"uses layers"| ModelShared
    Trainer -->|"computes losses via"| Losses
    Trainer -->|"writes checkpoints"| HFHub

    %% Inference Encode Flow
    EncodeCLI -->|"load pretrained"| HypencoderModel
    EncodeCLI -->|"tokenize via"| TokenizerUtils
    EncodeCLI -->|"encode with"| HypencoderModel
    EncodeCLI -->|"save embeddings"| JSONLUtils
    JSONLUtils -->|"store files"| IMGS

    %% Inference Retrieve Flow
    RetrieveCLI -->|"load embeddings via"| JSONLUtils
    RetrieveCLI -->|"build index"| NeighborGraph
    NeighborGraph -->|"approx search via"| ApproxRetrieve
    ApproxRetrieve -->|"score via"| QNet
    ApproxRetrieve -->|"evaluate via"| EvalUtils

    %% Shared Inference Utilties
    RetrieveCLI -->|"shared logic"| SharedInf
    EncodeCLI -->|"shared logic"| SharedInf

    %% External Library Dependencies
    HypencoderModel -->|"built on"| PyTorch
    QNet -->|"built on"| PyTorch
    Args -->|"uses"| HFHub
    DataCollator -->|"uses"| IRDatasets

    %% Click Events
    click TrainCLI "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/train/train.py"
    click EncodeCLI "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/inference/encode.py"
    click RetrieveCLI "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/inference/retrieve.py"
    click Args "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/train/args.py"
    click Configs "https://github.com/jfkback/hypencoder-paper/tree/main/hypencoder_cb/train/configs/"
    click DataCollator "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/train/data_collator.py"
    click Trainer "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/train/train.py"
    click ApproxRetrieve "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/inference/approx_retrieve.py"
    click NeighborGraph "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/inference/neighbor_graph.py"
    click SharedInf "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/inference/shared.py"
    click HypencoderModel "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/modeling/hypencoder.py"
    click QNet "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/modeling/q_net.py"
    click ModelShared "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/modeling/shared.py"
    click Losses "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/modeling/similarity_and_losses.py"
    click IOUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/io_utils.py"
    click JSONLUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/jsonl_utils.py"
    click EvalUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/eval_utils.py"
    click TokenizerUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/tokenizer_utils.py"
    click IteratorUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/iterator_utils.py"
    click TorchUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/torch_utils.py"
    click DataUtils "https://github.com/jfkback/hypencoder-paper/blob/main/hypencoder_cb/utils/data_utils.py"

    %% Styles
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef workflows fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef modeling fill:#90EE90,stroke:#333,stroke-width:1px
    classDef utils fill:#D3D3D3,stroke:#333,stroke-width:1px
```
