How jina-ai/late-chunking works
Project type and purpose
• This is a Python library and CLI tool for “late chunking” of long documents into smaller embedding vectors, plus an evaluation harness for retrieval tasks (BeIR/MTEB).
• It implements a context-sensitive chunking pipeline: apply a long-context transformer to full texts, then pool token embeddings into smaller chunks for vector search.
File structure
• Top-level:
• chunked_pooling/ ← core package
– chunking.py (implements token-level and chunk-level pooling)
– chunked_eval_tasks.py (defines evaluation tasks over BeIR/MTEB datasets)
– mteb_chunked_eval.py (wrapper for MTEB benchmarking)
– wrappers.py (helper functions to wrap embedding models, vector stores, metrics)
• run_chunked_eval.py ← CLI entry point that wires tasks, models, and metrics
• examples.ipynb ← demonstration notebook
• tests/ ← pytest suite (unit tests for API, chunking methods, v3)
• pyproject.toml ← build & dependency configuration
• .github/workflows/ci.yaml ← CI pipeline
README for additional insights
• Describes the “late chunking” method vs. traditional a priori chunking
• Outlines a two-phase pipeline:
1. full-text encoding into token embeddings via a long-context model (e.g., jina-embeddings-v2-base-en)
2. mean-pooling over sliding windows of token embeddings to produce chunk embeddings
• Evaluation on retrieval benchmarks (nDCG@10 on various datasets)
• CLI usage: python3 run_chunked_eval.py --task-name {TASK_NAME}
Detail is crucial—capture all essential elements
• High-level flow plus the major submodules (chunking.py vs. chunked_eval_tasks.py vs. wrappers.py)
• Where configuration (pyproject.toml) and CI hooks in
• Where tests validate functionality