# jina-ai/late-chunking architecture

> Architecture diagram of the GitHub repository jina-ai/late-chunking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jina-ai/late-chunking
- Repository: https://github.com/jina-ai/late-chunking

## Overview

##### Project type and purpose
- This is a Python library and CLI tool for “late chunking” of long documents into smaller embedding vectors, plus an evaluation harness for retrieval tasks (BeIR/MTEB).
- It implements a context-sensitive chunking pipeline: apply a long-context transformer to full texts, then pool token embeddings into smaller chunks for vector search.

##### File structure
- Top-level:
- chunked_pooling/    ← core package
– chunking.py           (implements token-level and chunk-level pooling)
– chunked_eval_tasks.py (defines evaluation tasks over BeIR/MTEB datasets)
– mteb_chunked_eval.py  (wrapper for MTEB benchmarking)
– wrappers.py           (helper functions to wrap embedding models, vector stores, metrics)
- run_chunked_eval.py  ← CLI entry point that wires tasks, models, and metrics
- examples.ipynb       ← demonstration notebook
- tests/               ← pytest suite (unit tests for API, chunking methods, v3)
- pyproject.toml       ← build & dependency configuration
- .github/workflows/ci.yaml ← CI pipeline

##### README for additional insights
- Describes the “late chunking” method vs. traditional a priori chunking
- Outlines a two-phase pipeline:
1. full-text encoding into token embeddings via a long-context model (e.g., jina-embeddings-v2-base-en)
2. mean-pooling over sliding windows of token embeddings to produce chunk embeddings
- Evaluation on retrieval benchmarks (nDCG@10 on various datasets)
- CLI usage: `python3 run_chunked_eval.py --task-name {TASK_NAME}`

##### Detail is crucial—capture all essential elements
- High-level flow plus the major submodules (chunking.py vs. chunked_eval_tasks.py vs. wrappers.py)
- Where configuration (pyproject.toml) and CI hooks in
- Where tests validate functionality

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Package Directory
    CorePackage["Core Package\n(chunked_pooling/)"]:::core

    subgraph "Core Library"
        Chunking["Chunking Module"]:::core
        EvalTasks["Evaluation Tasks Module"]:::core
        MTEBWrapper["MTEB Benchmark Wrapper"]:::core
        Wrappers["Wrappers (Model & Vector Store)"]:::core
        Pooling["Pooling Mechanism"]:::core
    end

    %% External Services
    InputDocs["Input Documents"]:::external
    QueryInput["Input Queries"]:::external
    EmbeddingModel["Embedding Model\n(Long-Context Transformer)"]:::external
    VectorIndex["Vector Database"]:::external
    Retrieval["Retrieval Engine"]:::external

    %% Orchestration
    CLI["CLI Driver"]:::orchestration
    Notebook["Demonstration Notebook"]:::orchestration

    %% Testing & CI
    subgraph "Test Suite" 
        TestsGroup["Test Harness"]:::testing
        API["test_api.py"]:::testing
        ChunkTests["test_chunking_methods.py"]:::testing
        V3Tests["test_v3.py"]:::testing
        Conftest["conftest.py"]:::testing
    end
    CI["CI Pipeline"]:::testing
    Config["Build & Dependency Config"]:::testing
    Readme["Project README"]:::testing

    %% Data Flow
    InputDocs -->|documents| Chunking
    Chunking -->|token encoding| EmbeddingModel
    EmbeddingModel -->|token embeddings| Pooling
    Pooling -->|chunk embeddings| VectorIndex
    VectorIndex -->|kNN search| Retrieval

    QueryInput -->|queries| EmbeddingModel
    EmbeddingModel -->|query tokens| Pooling
    Pooling -->|query embedding| Retrieval

    Retrieval -->|results| EvalTasks
    EvalTasks -->|benchmark tasks| MTEBWrapper
    MTEBWrapper -->|metrics| EvalTasks

    %% Orchestration Flows
    CLI --> Chunking
    CLI --> EmbeddingModel
    CLI --> Pooling
    CLI --> VectorIndex
    CLI --> Retrieval
    CLI --> EvalTasks

    Notebook --> Chunking
    Notebook --> Retrieval
    Notebook --> EvalTasks

    %% Testing & CI Flows
    TestsGroup --> Chunking
    TestsGroup --> EvalTasks
    TestsGroup --> Wrappers
    TestsGroup --> Pooling
    TestsGroup --> EmbeddingModel

    CI --> TestsGroup
    CI --> Config
    CI --> Readme

    %% Click Events
    click CorePackage "https://github.com/jina-ai/late-chunking/tree/main/chunked_pooling/"
    click Chunking "https://github.com/jina-ai/late-chunking/blob/main/chunked_pooling/chunking.py"
    click EvalTasks "https://github.com/jina-ai/late-chunking/blob/main/chunked_pooling/chunked_eval_tasks.py"
    click MTEBWrapper "https://github.com/jina-ai/late-chunking/blob/main/chunked_pooling/mteb_chunked_eval.py"
    click Wrappers "https://github.com/jina-ai/late-chunking/blob/main/chunked_pooling/wrappers.py"
    click CLI "https://github.com/jina-ai/late-chunking/blob/main/run_chunked_eval.py"
    click Notebook "https://github.com/jina-ai/late-chunking/blob/main/examples.ipynb"
    click TestsGroup "https://github.com/jina-ai/late-chunking/tree/main/tests/"
    click API "https://github.com/jina-ai/late-chunking/blob/main/tests/test_api.py"
    click ChunkTests "https://github.com/jina-ai/late-chunking/blob/main/tests/test_chunking_methods.py"
    click V3Tests "https://github.com/jina-ai/late-chunking/blob/main/tests/test_v3.py"
    click Conftest "https://github.com/jina-ai/late-chunking/blob/main/tests/conftest.py"
    click CI "https://github.com/jina-ai/late-chunking/blob/main/.github/workflows/ci.yaml"
    click Config "https://github.com/jina-ai/late-chunking/blob/main/pyproject.toml"
    click Readme "https://github.com/jina-ai/late-chunking/blob/main/README.md"

    %% Styles
    classDef core fill:#bbdefb,stroke:#0d47a1,color:#0d47a1;
    classDef external fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20;
    classDef orchestration fill:#ffe0b2,stroke:#e65100,color:#e65100;
    classDef testing fill:#e0e0e0,stroke:#424242,color:#424242;
```
