# indexnetwork/index architecture

> Architecture diagram of the GitHub repository indexnetwork/index, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/indexnetwork/index
- Repository: https://github.com/indexnetwork/index

## Overview

##### High-Level Architecture Components

a) Client Layer:
- Web App (Next.js based frontend)
- SDK implementations (JavaScript/Python)
- Widget integration

b) API Layer:
- REST API endpoints
- GraphQL interface
- Authentication/Authorization services

c) Core Protocol Layer:
- Indexer service
- Vector Database (ChromaDB)
- Agent-based query engine
- Natural Language Processing services

d) Storage Layer:
- Ceramic Network (ComposeDB)
- IPFS
- PostgreSQL

e) Integration Layer:
- Lit Protocol (access control)
- Apache Kafka (event streaming)
- External services (Zapier, etc.)

##### Component Relationships

a) Data Flow:
- Client requests → API Gateway → Core Services
- Indexer → Kafka → ChromaDB
- PostgreSQL CDC → Kafka Connect → Consumer Services
- Ceramic Network events → Indexer

b) Authentication Flow:
- DID-based authentication
- Lit Protocol integration
- Session management

##### Specific Implementation Details

Specialized components:

a) Indexer Architecture:
- Event consumers
- Vector embedding services
- Query processing
- Agent execution environment

b) Storage Architecture:
- Ceramic Network node
- IPFS node
- Vector store (ChromaDB)
- PostgreSQL database

##### Component Labels

Details in component labels:
- Component name
- Technology stack
- Primary responsibilities
- Key interfaces

##### Notes Section

Annotations for:
- Scaling considerations
- Security implementations
- Deployment requirements
- Performance optimizations

##### Security and access control mechanisms

This will provide a clear understanding of how Index Network functions as a decentralized discovery protocol while highlighting its key features: personalized information retrieval, composable vector database, and agent-based query engine.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Client
        WA[Web App]:::client
        JS[JavaScript SDK]:::client
        PY[Python SDK]:::client
        WI[Widget Integration]:::client
    end

    subgraph API
        REST[REST API]:::api
        GQL[GraphQL Interface]:::api
        AUTH[Authentication Services]:::security
    end

    subgraph Core
        IDX[Indexer Service]:::core
        VDB[Vector Database]:::core
        AQE[Agent Query Engine]:::core
        NLP[NLP Services]:::core
        CHAT[Chat System]:::core
    end

    subgraph Storage
        CDB[ComposeDB/Ceramic]:::storage
        IPFS[IPFS]:::storage
        PSQL[PostgreSQL]:::storage
    end

    subgraph Integration
        LIT[Lit Protocol]:::security
        KAFKA[Apache Kafka]:::integration
        EXT[External Services]:::integration
    end

    %% Client Layer Connections
    WA --> REST
    JS --> REST
    PY --> REST
    WI --> REST

    %% API Layer Connections
    REST --> IDX
    REST --> GQL
    GQL --> IDX
    AUTH --> LIT

    %% Core Layer Connections
    IDX --> VDB
    IDX --> KAFKA
    AQE --> VDB
    NLP --> AQE
    CHAT --> AQE

    %% Storage Layer Connections
    VDB --> PSQL
    CDB --> IDX
    IPFS --> CDB

    %% Integration Layer Connections
    LIT -.-> AUTH
    KAFKA -.-> IDX
    EXT -.-> REST

    %% Click Events
    click WA "https://github.com/indexnetwork/index/tree/dev/web-app"
    click JS "https://github.com/indexnetwork/index/tree/dev/sdk/js"
    click PY "https://github.com/indexnetwork/index/tree/dev/sdk/python"
    click WI "https://github.com/indexnetwork/index/tree/dev/widget"
    click REST "https://github.com/indexnetwork/index/tree/dev/api"
    click GQL "https://github.com/indexnetwork/index/blob/dev/api/src/packages/graphql-client.js"
    click AUTH "https://github.com/indexnetwork/index/tree/dev/api/src/middlewares"
    click IDX "https://github.com/indexnetwork/index/tree/dev/indexer"
    click VDB "https://github.com/indexnetwork/index/blob/dev/langchain/src/IndexVectorStore.ts"
    click AQE "https://github.com/indexnetwork/index/tree/dev/api/src/agents"
    click NLP "https://github.com/indexnetwork/index/tree/dev/api/src/language"
    click CHAT "https://github.com/indexnetwork/index/tree/dev/indexer/src/chat"
    click CDB "https://github.com/indexnetwork/index/blob/dev/api/src/controllers/composedb.js"
    click LIT "https://github.com/indexnetwork/index/tree/dev/api/src/libs/lit"
    click EXT "https://github.com/indexnetwork/index/blob/dev/api/src/controllers/zapier.js"

    classDef client fill:#2374ab
    classDef api fill:#48a9a6
    classDef core fill:#4ad66d
    classDef storage fill:#e4cc37
    classDef security fill:#d64933
    classDef integration fill:#8d89a6

    %% Legend
    subgraph Legend
        C[Client Components]:::client
        A[API Components]:::api
        CO[Core Components]:::core
        S[Storage Components]:::storage
        SE[Security Components]:::security
        I[Integration Components]:::integration
        
        %% Layout helpers for legend
        C --> A --> CO --> S --> SE --> I
    end
```
