# aurelio-labs/semantic-chunkers architecture

> Architecture diagram of the GitHub repository aurelio-labs/semantic-chunkers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aurelio-labs/semantic-chunkers
- Repository: https://github.com/aurelio-labs/semantic-chunkers

## Overview

##### Project type and purpose
- This is an open‑source Python library named “Semantic Chunkers” for intelligent, multi‑modal chunking of text, video, and audio.
- Its purpose is to provide a pluggable framework of chunking and splitting strategies to prepare content for downstream AI or data‑processing workflows.

##### File structure
- Top‑level directories and files:
– semantic_chunkers/           : main Python package
– docs/                        : user‑facing Jupyter notebooks and tutorials
– tests/                       : unit tests validating chunker and splitter behavior
– .github/workflows/           : CI/CD pipelines (linting, testing, release)
– pyproject.toml, Makefile     : build, dependency, and task definitions
- Within semantic_chunkers/:
– chunkers/                    : implementations of chunking strategies
- base.py                    : abstract base class
- consecutive.py, cumulative.py, regex.py, statistical.py
– splitters/                   : splitting strategies (base, regex)
– utils/                       : helpers (logging, text transformations)
– schema.py                    : data models or type schemas for chunker inputs/outputs

##### README for additional insights
- Key features: multi‑modal chunking, AI‑centric, extensible via custom chunkers/splitters

##### Detail without overthinking
- Break the library into as many logical components as needed to capture its extensible, modular nature
- Even small utilities (logger, text helpers) if they play a recurring role in the flow

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Public Interface
    subgraph "Public Interface"
        API["Public API"]:::core
    end

    %% Core Framework
    subgraph "Core Framework"
        FactoryC["ChunkerFactory"]:::core
        FactoryS["SplitterFactory"]:::core
        BaseC["BaseChunker"]:::core
        BaseS["BaseSplitter"]:::core
    end

    %% Built-in Implementations
    subgraph "Built-in Chunkers"
        Consecutive["ConsecutiveChunker"]:::impl
        Cumulative["CumulativeChunker"]:::impl
        RegexC["RegexChunker"]:::impl
        Statistical["StatisticalChunker"]:::impl
    end

    subgraph "Built-in Splitters"
        RegexS["RegexSplitter"]:::impl
    end

    %% Utilities
    subgraph "Utilities"
        Logger["Logger"]:::util
        TextHelper["TextHelpers"]:::util
    end

    %% Data Models
    subgraph "Data Models"
        Schema["Schema"]:::datamodel
    end

    %% Documentation
    subgraph "Documentation"
        NB1["00-chunkers-intro.ipynb"]:::docs
        NB2["01-video-chunking.ipynb"]:::docs
        NB3["02-chunkers-async.ipynb"]:::docs
    end

    %% Tests
    subgraph "Tests"
        T1["test_chunkers.py"]:::tests
        T2["test_regex_chunker.py"]:::tests
        T3["test_regex_splitter.py"]:::tests
    end

    %% CI/CD Pipelines
    subgraph "CI/CD Pipelines"
        Lint["lint.yml"]:::pipelines
        TestY["test.yml"]:::pipelines
        Release["release.yml"]:::pipelines
        Commits["conventional_commits.yml"]:::pipelines
        PRAgent["pr_agent.yml"]:::pipelines
    end

    %% Build & Packaging
    subgraph "Build & Packaging"
        Ptoml["pyproject.toml"]:::external
        Make["Makefile"]:::external
    end

    %% Relationships
    API --> FactoryC
    API --> FactoryS

    FactoryC --> BaseC
    FactoryS --> BaseS

    FactoryC --> Consecutive
    FactoryC --> Cumulative
    FactoryC --> RegexC
    FactoryC --> Statistical

    FactoryS --> RegexS

    Consecutive --> RegexS
    Cumulative --> RegexS
    RegexC --> RegexS
    Statistical --> RegexS

    Logger --> Consecutive
    Logger --> Cumulative
    Logger --> RegexC
    Logger --> Statistical
    Logger --> RegexS

    TextHelper --> Consecutive
    TextHelper --> Cumulative
    TextHelper --> RegexC
    TextHelper --> Statistical
    TextHelper --> RegexS

    Schema --> FactoryC
    Schema --> FactoryS
    Schema --> Consecutive
    Schema --> Cumulative
    Schema --> RegexC
    Schema --> Statistical
    Schema --> RegexS

    Lint --> TestY
    TestY --> T1
    TestY --> T2
    TestY --> T3
    Release --> Ptoml
    Release --> Make

    %% Click Events
    click API "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/__init__.py"
    click BaseC "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/chunkers/base.py"
    click BaseS "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/splitters/base.py"
    click Consecutive "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/chunkers/consecutive.py"
    click Cumulative "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/chunkers/cumulative.py"
    click RegexC "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/chunkers/regex.py"
    click Statistical "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/chunkers/statistical.py"
    click RegexS "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/splitters/regex.py"
    click Logger "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/utils/logger.py"
    click TextHelper "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/utils/text.py"
    click Schema "https://github.com/aurelio-labs/semantic-chunkers/blob/main/semantic_chunkers/schema.py"
    click NB1 "https://github.com/aurelio-labs/semantic-chunkers/blob/main/docs/00-chunkers-intro.ipynb"
    click NB2 "https://github.com/aurelio-labs/semantic-chunkers/blob/main/docs/01-video-chunking.ipynb"
    click NB3 "https://github.com/aurelio-labs/semantic-chunkers/blob/main/docs/02-chunkers-async.ipynb"
    click T1 "https://github.com/aurelio-labs/semantic-chunkers/blob/main/tests/unit/test_chunkers.py"
    click T2 "https://github.com/aurelio-labs/semantic-chunkers/blob/main/tests/unit/test_regex_chunker.py"
    click T3 "https://github.com/aurelio-labs/semantic-chunkers/blob/main/tests/unit/test_regex_splitter.py"
    click Lint "https://github.com/aurelio-labs/semantic-chunkers/blob/main/.github/workflows/lint.yml"
    click TestY "https://github.com/aurelio-labs/semantic-chunkers/blob/main/.github/workflows/test.yml"
    click Release "https://github.com/aurelio-labs/semantic-chunkers/blob/main/.github/workflows/release.yml"
    click Commits "https://github.com/aurelio-labs/semantic-chunkers/blob/main/.github/workflows/conventional_commits.yml"
    click PRAgent "https://github.com/aurelio-labs/semantic-chunkers/blob/main/.github/workflows/pr_agent.yml"
    click Ptoml "https://github.com/aurelio-labs/semantic-chunkers/blob/main/pyproject.toml"
    click Make "https://github.com/aurelio-labs/semantic-chunkers/tree/main/Makefile"

    %% Styles
    classDef core fill:#D6EAF8,stroke:#1B4F72
    classDef impl fill:#D5F5E3,stroke:#186A3B
    classDef util fill:#FCF3CF,stroke:#B7950B
    classDef datamodel fill:#E8DAEF,stroke:#76448A
    classDef docs fill:#EBDEF0,stroke:#6C3483
    classDef tests fill:#E8F8F5,stroke:#117A65
    classDef pipelines fill:#F5EEF8,stroke:#7D3C98
    classDef external fill:#FADBD8,stroke:#922B21
```
