# aryn-ai/sycamore architecture

> Architecture diagram of the GitHub repository aryn-ai/sycamore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aryn-ai/sycamore
- Repository: https://github.com/aryn-ai/sycamore

## Overview

##### Main Components
- Core Document Processing Engine: Represent the “lib/sycamore” module that implements the DocSet abstraction, core data structures, utilities, and transformation logic (including various transforms and functions for cleaning, enriching, and processing documents). This is the central “brain” of the system.
- Document Ingestion & Transformation: Show how input documents (e.g., PDFs, images, HTML, etc.) are input into the system, processed (using segmentation provided by Aryn DocParse) and enriched. Consider including the “lib/aryn-sdk” (which acts as a client to the Aryn partitioning service) as a sub-component.
- Remote Processor Service: Include the “apps/remote-processor-service” and the corresponding “lib/remote-processors.” These components are responsible for handling distributed processing and scaling using tools like Ray.
- Integration and Testing: Include the “apps/integration” module which demonstrates usage and supports automated testing or orchestration through scripts.
- Developer and Interactive Environment: Represent the “apps/jupyter” component that provides an interactive development environment for iteratively processing documents and prototyping ETL jobs.

##### Relationships and Data Flow
- Input documents are received via one or more ingestion interfaces. Once ingested, documents pass through the core processing engine (DocSet) where they are partitioned, transformed, and enriched.
- The processing engine uses the Aryn DocParse API (or a local version when running in Docker) to perform complex document segmentation.
- Remote or distributed processing is managed by the remote processor service, which coordinates processing workloads across multiple workers. Indicate this layer as communicating with the core processing engine.
- Integration tests and automation scripts as a feedback loop, ensuring each component interacts correctly.

##### Architectural Patterns and Design Principles
- Modular design with clearly separated concerns: document ingestion, processing/enrichment, and data loading.
- Abstraction provided by DocSets that encapsulate functional programming techniques for transforming unstructured data.
- Scalable architecture enabled by Dockerized services and Ray (for distributed processing), which also supports microservice patterns for remote processing.

##### Technologies and Frameworks
- Docker: Represent Docker components as containers (apps/docker-base, apps/jupyter, apps/opensearch, apps/remote-processor-service) that encapsulate runtime environments.
- Python: Identify that the core engine, connectors, and transformation functions are implemented in Python.
- External APIs: Emphasize the connection to the Aryn DocParse service, which is a key dependency for document segmentation.
- Vector Databases: Indicate that multiple vector DB connectors are available as “pluggable modules.”

## Diagram (Mermaid)

```mermaid
graph TD
    DOC_ING("Document Ingestion (PDFs,Images,HTML)"):::appStyle
    ARYN("Aryn SDK Interface"):::coreLib
    CORE("Core Processing Engine (DocSet)"):::coreLib
    API("Aryn DocParse API"):::external
    CONNECT("Connectors Layer"):::coreLib
    VECTOR("Vector Databases (OpenSearch,ElasticSearch,Pinecone,DuckDB,Qdrant,Weaviate)"):::external
    REMOTE("Remote Processor Service"):::appStyle
    RAY("Ray Distributed Processing"):::external
    INTEGRATION("Integration & Testing"):::appStyle
    JUPYTER("Jupyter Environment"):::appStyle

    DOC_ING -->|"sendsData"| ARYN
    ARYN -->|"processesData"| CORE
    CORE -->|"calls"| API
    CORE -->|"loadsData"| CONNECT
    CONNECT -->|"updates"| VECTOR
    CORE -->|"delegates"| REMOTE
    REMOTE -->|"scalesWith"| RAY
    INTEGRATION -->|"tests"| CORE
    INTEGRATION -->|"tests"| REMOTE
    JUPYTER -->|"interactive"| CORE

    click CORE "https://github.com/aryn-ai/sycamore/tree/main/lib/sycamore/sycamore"
    click ARYN "https://github.com/aryn-ai/sycamore/tree/main/lib/aryn-sdk"
    click CONNECT "https://github.com/aryn-ai/sycamore/tree/main/lib/sycamore/sycamore/connectors"
    click REMOTE "https://github.com/aryn-ai/sycamore/tree/main/apps/remote-processor-service"
    click INTEGRATION "https://github.com/aryn-ai/sycamore/tree/main/apps/integration"
    click JUPYTER "https://github.com/aryn-ai/sycamore/tree/main/apps/jupyter"

    classDef coreLib fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef appStyle fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef external fill:#F5F5DC,stroke:#000,stroke-width:2px;
```
