# mantisai/sieves architecture

> Architecture diagram of the GitHub repository mantisai/sieves, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mantisai/sieves
- Repository: https://github.com/mantisai/sieves

## Overview

##### Project Type and Purpose
- Type: A library/toolkit (not a full-stack app) for zero- and few-shot NLP using structured generation.
- Purpose: Provide a unified, document-based pipeline for preprocessing, prompting LLMs (via multiple backends), and postprocessing, all with caching, serialization, and observability.

##### File Structure
- Top-level modules:
– sieves/data: document abstractions and parsers
– sieves/engines: core Engine class, wrappers, and adapter “Bridge” imports for each backend (dspy, gliner, huggingface, etc.)
– sieves/pipeline: pipeline orchestration and execution logic
– sieves/tasks: individual Task classes grouped by preprocessing, predictive, and postprocessing
– sieves/serialization.py: pipeline and Doc persistence
- Configuration/build: pyproject.toml, setup.py, mkdocs.yml, GitHub workflows, tests.
- Docs: extensive mkdocs-based documentation covering guides, pipelines, engines, tasks.

##### README for Architectural Insights
- Core abstractions: Pipeline, Doc, Task, Engine, Bridge
- Pattern: pipeline of tasks, each task instantiates a Bridge to connect to an Engine, which in turn standardizes calls to various structured-generation backends.
- Features: caching, serialization, extensibility for new tasks/engines.

##### Main Components
a. Pipeline (orchestrator)
b. Doc (data carrier)
c. Task nodes (preprocessing, predictive, postprocessing)
d. Bridge adapters (engine-specific prompt templates and type handlers)
e. Engine abstraction (unified interface)
f. External Model Backends (dspy, outlines, huggingface transformers, instructor, ollama, etc.)
g. Caching layer (prevents redundant calls)
h. Serialization store (pipeline.yml, pickled Docs)
i. Optional utilities: chunkers (chonkie), parsers (docling, marker, unstructured)

##### Relationships and Interactions
- Within Pipeline: for each Task:
Doc → Task → Bridge → Engine → Model Backend → Engine → Bridge → Task → Doc.results
- Caching: sits between Bridge/Engine and backends, intercepts requests/responses
- Serialization: Pipeline dump/load connects to disk storage (YAML + pickled state)

##### Architectural Patterns and Principles
- Pipeline pattern: sequential, configurable task graph
- Adapter/Bridge pattern: decouple Tasks from diverse backends via Bridge → Engine
- Single source of truth: Doc carries state/results
- Modular/extensible: plugin points in sieves/tasks and sieves/engines
- Separation of concerns: parsing, chunking, prompting, distillation, serialization

##### Technologies, Frameworks, Libraries
- Core: Python 3, pytest, black, mkdocs
- External libs: docling, unstructured, marker, chonkie, LangChain, HuggingFace Transformers, dspy, ollama, outlines
- CI/CD: GitHub Actions, codecov
- Packaging: setuptools, pyproject.toml

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Library
    subgraph "Core Library"
        Pipeline["Pipeline\n(sieves/pipeline/core.py)"]:::core
        Doc["Doc\n(sieves/data/doc.py)"]:::data
        Ser["Serialization\n(sieves/serialization.py)"]:::storage
        Cache["Cache Store"]:::storage

        subgraph "Tasks"
            TaskBase["Task (base)\n(sieves/tasks/core.py)"]:::task

            subgraph "Preprocessing"
                PreMod["Preprocessing tasks\n(sieves/tasks/preprocessing/)"]:::task
                Chunkers["Chunking utils\n(sieves/tasks/preprocessing/chunking/)"]:::util
                OCR["OCR/Parsers\n(sieves/tasks/preprocessing/ocr/)"]:::util
            end

            subgraph "Predictive"
                PredMod["Predictive tasks\n(sieves/tasks/predictive/)"]:::task
            end

            subgraph "Postprocessing"
                PostMod["Postprocessing tasks\n(sieves/tasks/postprocessing/)"]:::task
            end
        end

        Engine["Engine\n(sieves/engines/core.py)"]:::engine
        BridgeReg["Bridge Registry\n(sieves/engines/engine_import.py)"]:::engine
        BridgeWrap["Bridge Wrapper\n(sieves/engines/wrapper.py)"]:::engine
    end

    %% External Model Backends
    subgraph "External Model Backends"
        dspy["dspy"]:::cloud
        glix["gliner"]:::cloud
        hugging["HuggingFace"]:::cloud
        instructor["Instructor"]:::cloud
        langchain["LangChain"]:::cloud
        ollama["Ollama"]:::cloud
        outlines["Outlines"]:::cloud
        vllm["vLLM"]:::cloud
    end

    %% Data Flow
    Pipeline -->|loads| Doc
    Pipeline -->|orchestrates| TaskBase
    Pipeline -->|serialize/load| Ser
    Pipeline -->|uses| Cache

    Doc -->|passes to| TaskBase

    TaskBase -->|routes to| PreMod
    TaskBase -->|routes to| PredMod
    TaskBase -->|routes to| PostMod

    PreMod --> Chunkers
    PreMod --> OCR

    TaskBase --> BridgeReg
    BridgeReg --> BridgeWrap
    BridgeWrap -->|calls| Engine
    Engine -->|invokes| dspy
    Engine -->|invokes| glix
    Engine -->|invokes| hugging
    Engine -->|invokes| instructor
    Engine -->|invokes| langchain
    Engine -->|invokes| ollama
    Engine -->|invokes| outlines
    Engine -->|invokes| vllm
    BridgeWrap -->|returns results| TaskBase
    TaskBase -->|updates| Doc

    %% Click Events
    click Pipeline "https://github.com/mantisai/sieves/blob/main/sieves/pipeline/core.py"
    click Doc "https://github.com/mantisai/sieves/blob/main/sieves/data/doc.py"
    click TaskBase "https://github.com/mantisai/sieves/blob/main/sieves/tasks/core.py"
    click PreMod "https://github.com/mantisai/sieves/tree/main/sieves/tasks/preprocessing/"
    click PredMod "https://github.com/mantisai/sieves/tree/main/sieves/tasks/predictive/"
    click PostMod "https://github.com/mantisai/sieves/tree/main/sieves/tasks/postprocessing/"
    click Engine "https://github.com/mantisai/sieves/blob/main/sieves/engines/core.py"
    click BridgeReg "https://github.com/mantisai/sieves/blob/main/sieves/engines/engine_import.py"
    click BridgeWrap "https://github.com/mantisai/sieves/blob/main/sieves/engines/wrapper.py"
    click dspy "https://github.com/mantisai/sieves/blob/main/sieves/engines/dspy_.py"
    click glix "https://github.com/mantisai/sieves/blob/main/sieves/engines/glix_.py"
    click hugging "https://github.com/mantisai/sieves/blob/main/sieves/engines/huggingface_.py"
    click instructor "https://github.com/mantisai/sieves/blob/main/sieves/engines/instructor_.py"
    click langchain "https://github.com/mantisai/sieves/blob/main/sieves/engines/langchain_.py"
    click ollama "https://github.com/mantisai/sieves/blob/main/sieves/engines/ollama_.py"
    click outlines "https://github.com/mantisai/sieves/blob/main/sieves/engines/outlines_.py"
    click vllm "https://github.com/mantisai/sieves/blob/main/sieves/engines/vllm_.py"

    %% Styles
    classDef core fill:#D6E9C6,stroke:#3C763D,color:#3C763D;
    classDef data fill:#B3E5FC,stroke:#0277BD,color:#01579B;
    classDef task fill:#FFF3CD,stroke:#856404,color:#856404;
    classDef engine fill:#D1ECF1,stroke:#0C5460,color:#0C5460;
    classDef storage fill:#F5C6CB,stroke:#721C24,color:#721C24;
    classDef cloud fill:#E2E3E5,stroke:#6C757D,color:#6C757D;
    classDef util fill:#E8EAF6,stroke:#3F51B5,color:#303F9F;
```
