# cocoindex-io/cocoindex architecture

> Architecture diagram of the GitHub repository cocoindex-io/cocoindex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cocoindex-io/cocoindex
- Repository: https://github.com/cocoindex-io/cocoindex

## Overview

##### Project type and purpose
- CocoIndex is primarily an open-source data transformation framework (library + CLI + optional server) whose core engine is implemented in Rust and exposed via a Python API/CLI.
- It implements a dataflow programming model for incremental, real-time ingestion, transformation, and indexing of data (e.g., embeddings, knowledge graphs) aimed at powering AI use cases.

##### File structure
- Top-level “src” directory: Rust core engine, organized into sub-modules (builder, execution, ops, llm, service, setup, utils).
- “python/cocoindex”: Python package exposing the Rust core via pyo3/FFI—including CLI, FlowBuilder, user-facing APIs, tests.
- “examples”: standalone demo flows in Python illustrating how to drive the engine (local files, S3, GDrive, FastAPI server, etc.).
- “dev” & “.github”: Docker/Kubernetes dev manifests, CI/CD workflows, release scripts.
- “docs” & “README.md”: Docusaurus site and onboarding guides.

##### README for additional insights
- Describes core features: dataflow programming, change-data capture, incremental processing, exporting to relational/graph/vector stores.
- Lists supported sources (LocalFile, S3, Google Drive), storages (Postgres, Neo4j, Qdrant), LLM providers (OpenAI, Anthropic, Gemini, Ollama).

##### Main components
a. User layer (Python client/CLI/Server)
- Python SDK (python/cocoindex)
- CLI entrypoint
- Optional HTTP server (src/server.rs + service modules)
b. Core engine (Rust)
- FlowBuilder & DataScope (builder module)
- Execution engine (execution module): orchestrates source reads, transforms, collectors
- Ops & Functions (ops/functions, ops/sources, ops/storages): pluggable connectors
- LLM adapters (llm module): HTTP clients to OpenAI, Anthropic, etc.
- Incremental state & metadata (setup & utils modules): change-data capture, lineage, fingerprinting
c. External dependencies/services
- Source systems: Local FS, Amazon S3, Google Drive, databases, whatever user configures
- LLM APIs: OpenAI, Anthropic, Gemini, Ollama
- Persistence/Index targets:
- Postgres (for incremental state & vector storage)
- Neo4j (graph storage)
- Qdrant (vector storage)
- Other supported storages (shared modules)

##### Relationships and interactions
- User writes Python flow definitions → invokes SDK/CLI → Python layer calls Rust core via pyo3.
- Rust builder constructs an in-memory DAG of transformations (sources → transforms → collectors).
- Execution engine schedules and runs each node:
- Reads from sources
- Applies functions (pure transforms, LLM calls)
- Tracks change sets for incremental updates
- Emits rows to collectors
- Collector.export pushes data to configured storages via ops/storages plugins.
- Optional Rust HTTP server layer exposes endpoints to trigger flows or perform searches via service modules.

##### Architectural patterns and design principles
- Dataflow programming: each transform is a node in a DAG, no hidden state, immutable per-row data lineage.
- Incremental processing / CDC: engine determines minimal updates based on change logs.
- Plugin architecture: operations, sources, storages, LLM adapters are defined by trait-based interfaces + dynamic registry.
- Polyglot API: Rust core + Python bindings via pyo3, enabling performance with ease of use.

##### Key elements to capture
- Immutable Dataflow DAG: sources → transforms → collectors
- Incremental state manager tracking deltas in Postgres
- Plugin registry for dynamic ops discovery
- Polyglot boundary (Python ↔ Rust)
- External integrations (LLM, storage, sources)

##### Detail without overthinking
- Break the system into as many components as makes sense: Python layer, Rust engine, builder, executor, plugins, external systems

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    subgraph "User Layer"
        PythonSDK["Python SDK"]:::python
        CLI["CLI Entrypoint"]:::python
        HTTPServer["Optional HTTP Server"]:::python
    end

    %% Rust Core Engine
    subgraph "Rust Core Engine"
        Builder["FlowBuilder & DataScope"]:::rust
        Exec["Execution Engine"]:::rust
        Ops["Ops & Functions"]:::rust
        LLM["LLM Adapters"]:::rust
        State["Incremental State & Metadata"]:::rust
        FFI["PyO3 Bindings"]:::rust
        CoreLib["Core Library Entrypoint"]:::rust
    end

    %% External Sources
    subgraph "External Data Sources"
        LocalFS["Local File System"]:::ext
        S3["Amazon S3"]:::ext
        GDrive["Google Drive"]:::ext
    end

    %% LLM Services
    subgraph "LLM Services"
        OpenAI["OpenAI API"]:::llm
        Anthropic["Anthropic API"]:::llm
        Gemini["Gemini API"]:::llm
        Ollama["Ollama API"]:::llm
    end

    %% Datastores
    subgraph "Storage Targets"
        Postgres["Postgres"]:::db
        Neo4j["Neo4j"]:::db
        Qdrant["Qdrant"]:::db
        Kuzu["Kuzu"]:::db
    end

    %% Dataflow DAG (conceptual)
    Builder --> Exec
    Exec --> SourcesOps["Sources / Transforms / Collectors"]:::rust
    SourcesOps --> LocalFS
    SourcesOps --> S3
    SourcesOps --> GDrive
    Exec --> LLM
    Exec --> State
    Exec --> DatastoresOps["Storages"]:::rust
    DatastoresOps --> Postgres
    DatastoresOps --> Neo4j
    DatastoresOps --> Qdrant
    DatastoresOps --> Kuzu

    %% Polyglot boundary
    PythonSDK --> FFI
    CLI --> PythonSDK
    HTTPServer --> CoreLib
    FFI --> CoreLib
    CoreLib --> Builder

    %% Connections to external services
    LLM --> OpenAI
    LLM --> Anthropic
    LLM --> Gemini
    LLM --> Ollama

    %% Click Events
    click PythonSDK "https://github.com/cocoindex-io/cocoindex/tree/main/python/cocoindex/"
    click CLI "https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/cli.py"
    click HTTPServer "https://github.com/cocoindex-io/cocoindex/blob/main/src/server.rs"
    click Builder "https://github.com/cocoindex-io/cocoindex/tree/main/src/builder/"
    click Exec "https://github.com/cocoindex-io/cocoindex/tree/main/src/execution/"
    click Ops "https://github.com/cocoindex-io/cocoindex/tree/main/src/ops/"
    click LLM "https://github.com/cocoindex-io/cocoindex/tree/main/src/llm/"
    click State "https://github.com/cocoindex-io/cocoindex/tree/main/src/setup/"
    click FFI "https://github.com/cocoindex-io/cocoindex/tree/main/src/py/"
    click CoreLib "https://github.com/cocoindex-io/cocoindex/blob/main/src/lib.rs"

    %% Styles
    classDef python fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef rust fill:#FFE0B2,stroke:#e65100,color:#e65100
    classDef ext fill:#E8F5E9,stroke:#2e7d32,color:#2e7d32
    classDef llm fill:#F3E5F5,stroke:#6a1b9a,color:#6a1b9a
    classDef db fill:#FFEBEE,stroke:#c62828,color:#c62828
```
