How cocoindex-io/cocoindex works
Project type and purpose
• CocoIndex is primarily an open-source data transformation framework (library + CLI + optional server) whose core engine is implemented in Rust and exposed via a Python API/CLI.
• It implements a dataflow programming model for incremental, real-time ingestion, transformation, and indexing of data (e.g., embeddings, knowledge graphs) aimed at powering AI use cases.
File structure
• Top-level “src” directory: Rust core engine, organized into sub-modules (builder, execution, ops, llm, service, setup, utils).
• “python/cocoindex”: Python package exposing the Rust core via pyo3/FFI—including CLI, FlowBuilder, user-facing APIs, tests.
• “examples”: standalone demo flows in Python illustrating how to drive the engine (local files, S3, GDrive, FastAPI server, etc.).
• “dev” & “.github”: Docker/Kubernetes dev manifests, CI/CD workflows, release scripts.
• “docs” & “README.md”: Docusaurus site and onboarding guides.
README for additional insights
• Describes core features: dataflow programming, change-data capture, incremental processing, exporting to relational/graph/vector stores.
• Lists supported sources (LocalFile, S3, Google Drive), storages (Postgres, Neo4j, Qdrant), LLM providers (OpenAI, Anthropic, Gemini, Ollama).
Main components
a. User layer (Python client/CLI/Server)
• Python SDK (python/cocoindex)
• CLI entrypoint
• Optional HTTP server (src/server.rs + service modules)
b. Core engine (Rust)
• FlowBuilder & DataScope (builder module)
• Execution engine (execution module): orchestrates source reads, transforms, collectors
• Ops & Functions (ops/functions, ops/sources, ops/storages): pluggable connectors
• LLM adapters (llm module): HTTP clients to OpenAI, Anthropic, etc.
• Incremental state & metadata (setup & utils modules): change-data capture, lineage, fingerprinting
c. External dependencies/services
• Source systems: Local FS, Amazon S3, Google Drive, databases, whatever user configures
• LLM APIs: OpenAI, Anthropic, Gemini, Ollama
• Persistence/Index targets:
• Postgres (for incremental state & vector storage)
• Neo4j (graph storage)
• Qdrant (vector storage)
• Other supported storages (shared modules)
Relationships and interactions
• User writes Python flow definitions → invokes SDK/CLI → Python layer calls Rust core via pyo3.
• Rust builder constructs an in-memory DAG of transformations (sources → transforms → collectors).
• Execution engine schedules and runs each node:
• Reads from sources
• Applies functions (pure transforms, LLM calls)
• Tracks change sets for incremental updates
• Emits rows to collectors
• Collector.export pushes data to configured storages via ops/storages plugins.
• Optional Rust HTTP server layer exposes endpoints to trigger flows or perform searches via service modules.
Architectural patterns and design principles
• Dataflow programming: each transform is a node in a DAG, no hidden state, immutable per-row data lineage.
• Incremental processing / CDC: engine determines minimal updates based on change logs.
• Plugin architecture: operations, sources, storages, LLM adapters are defined by trait-based interfaces + dynamic registry.
• Polyglot API: Rust core + Python bindings via pyo3, enabling performance with ease of use.
Key elements to capture
• Immutable Dataflow DAG: sources → transforms → collectors
• Incremental state manager tracking deltas in Postgres
• Plugin registry for dynamic ops discovery
• Polyglot boundary (Python ↔ Rust)
• External integrations (LLM, storage, sources)
Detail without overthinking
• Break the system into as many components as makes sense: Python layer, Rust engine, builder, executor, plugins, external systems