# anyproto/any-sync architecture

> Architecture diagram of the GitHub repository anyproto/any-sync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anyproto/any-sync
- Repository: https://github.com/anyproto/any-sync

## Overview

##### Project Type & Purpose
- This is a backend “sync node” service implementing the Any-Sync protocol in Go.
- It exposes RPC endpoints (via dRPC) for clients and peers, manages spaces (CRDT-based graph databases), files, ACLs, consensus, coordinator, nameservice, identity, and payments.
- It is a modular monolith broken into service components and network/storage layers.

##### Core Layers & Directories

A. “Edge” / Clients Layer
- Anytype UI or other local applications (SDKs)
- Peer Sync Peers (other Any-Sync Nodes)

B. RPC & Network Layer (net/)
- net/transport (QUIC, Yamux) for raw multiplexed streams
- net/streampool & net/pool for connection & stream management
- net/rpc (drpc server & client) for RPC dispatch
- net/secureservice for handshake & E2E encryption
- net/peerservice for peer discovery & connection pooling

C. Service & Protocol Layer
- commonspace/spaceservice: space CRDT orchestration (head sync, object sync, tree sync, deletion, settings)
- commonfile/fileservice: file block storage & file sync
- accountservice: account setup & authorization
- acl: access control list service
- consensus/consensusclient: watch ACL changes via consensus node
- coordinator/coordinatorclient: retrieve network configuration from coordinator node
- nameservice/nameserviceclient: DNS-like discovery
- identityrepo: identity management proto
- paymentservice: quota & payment tiers
- metric: metrics collection & reporting

D. Core Engine & Utilities
- commonspace/* submodules (object tree builder, object manager, peer manager, deletion manager)
- app/ (ldiff vs olddiff, logger, ocache, debugstat) for local caching, logging, and diffing
- util/ (crypto, syncqueues, multiqueue, periodicsync, storeutil)

E. Storage Layer
- commonspace/spacestorage (migration, oldstorage, new storage)
- commonfile/fileblockstore
- nodeconf/nodeconfstore for node configuration persistence
- util/crypto keystorage

##### Data Flows & Interactions
- Clients → dRPC Server (net/rpc) → Service Handlers → Core Engine → Storage → back up the same chain.
- Peer Sync: Service Handlers invoke commonspace/sync → net/secureservice → net/transport → Peer → reverse path.
- External Dependencies: services call coordinatorclient, consensusclient, nameserviceclient via dRPC or DRPC-proto stubs.

##### Architectural Patterns & Design Principles
- Layered architecture with clear separation (Network → Service → Engine → Storage).
- Modular services, each with its own proto definitions and client stubs.
- Dependency injection & mocking heavy for testability (see mock_* packages everywhere).
- CRDT pipelines: head sync → object sync → tree sync, each as a separate flow.
- Use of multiplexed connections and streams (Yamux/QUIC) under the hood.

##### Technologies & Frameworks
- Go modules and package structure.
- Protocols: Protocol Buffers & dRPC.
- Transport: QUIC, Yamux, TCP.
- Crypto: ed25519, x25519, AES, user-controlled keys.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Edge / Clients Layer
    subgraph "Edge / Clients Layer"
        direction BT
        Client["Anytype UI / SDKs"]:::external
        PeerNode["Sync Peer Node"]:::external
    end

    %% RPC & Network Layer
    subgraph "RPC & Network Layer"
        direction BT
        Transport["net/transport"]:::network
        StreamPool["net/streampool"]:::network
        ConnPool["net/pool"]:::network
        RPC["net/rpc"]:::network
        SecureService["net/secureservice"]:::network
        PeerService["net/peerservice"]:::network
    end

    %% Service & Protocol Layer
    subgraph "Service & Protocol Layer"
        direction BT
        SpaceService["commonspace/spaceservice.go"]:::service
        FileService["commonfile/fileservice/fileservice.go"]:::service
        BlockService["commonfile/fileservice/blockservice.go"]:::service
        AccountService["accountservice/accountservice.go"]:::service
        ACLService["acl/acl.go"]:::service
        ConsensusClient["consensus/consensusclient/client.go"]:::service
        CoordinatorClient["coordinator/coordinatorclient/coordinatorclient.go"]:::service
        NameServiceClient["nameservice/nameserviceclient/nameserviceclient.go"]:::service
        IdentityRepo["identityrepo/identityrepoproto/identityrepo.pb.go"]:::service
        PaymentService["paymentservice/paymentserviceclient/paymentserviceclient.go"]:::service
        MetricService["metric/metric.go"]:::service
    end

    %% Core Engine & Utilities
    subgraph "Core Engine & Utilities"
        direction BT
        App["app/app.go"]:::core
        Logger["app/logger/log.go"]:::core
        OCache["app/ocache/ocache.go"]:::core
        DebugStat["app/debugstat/stat.go"]:::core
        LDiff["app/ldiff/diff.go"]:::core
        OldDiff["app/olddiff/diff.go"]:::core
        ObjManager["commonspace/objectmanager/objectmanager.go"]:::core
        TreeBuilder["commonspace/objecttreebuilder/treebuilder.go"]:::core
        PeerManager["commonspace/peermanager/peermanager.go"]:::core
        DeletionManager["commonspace/deletionmanager/deletionmanager.go"]:::core
        SyncCore["commonspace/sync/sync.go"]:::core
        Utilities["util/*"]:::core
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction BT
        SpaceStorage["commonspace/spacestorage/spacestorage.go"]:::storage
        FileBlockStore["commonfile/fileblockstore/fileblockstore.go"]:::storage
        NodeConfStore["nodeconf/nodeconfstore/nodeconfstore.go"]:::storage
        KeyStorage["util/crypto/keystorage.go"]:::storage
    end

    %% Connections: Clients to RPC
    Client -->|"RPC Call"| RPC
    PeerNode -->|"Peer Connect"| PeerService

    %% RPC internals
    RPC -->|"uses"| Transport
    RPC -->|"manages streams"| StreamPool
    RPC -->|"manages conns"| ConnPool
    RPC -->|"dispatches to"| SecureService
    SecureService -->|"secures traffic"| Transport

    %% RPC to Services
    RPC -->|"FetchSpace"| SpaceService
    RPC -->|"FileOps"| FileService
    RPC -->|"AccountOps"| AccountService
    RPC -->|"ACLOps"| ACLService
    RPC -->|"PaymentCheck"| PaymentService
    RPC -->|"MetricsReport"| MetricService

    %% Service to Core
    SpaceService -->|"SyncRequest"| SyncCore
    FileService -->|"StoreBlock"| BlockService
    AccountService -->|"LoadConfig"| NodeConfStore
    ACLService -->|"Verify"| ObjManager
    ConsensusClient -->|"WatchACL"| SpaceService
    CoordinatorClient -->|"GetConfig"| SpaceService
    NameServiceClient -->|"DiscoverPeers"| PeerService
    IdentityRepo -->|"Identify"| SecureService
    PaymentService -->|"QuotaCheck"| SpaceService
    MetricService -->|"Collect"| Logger

    %% Core internal flows
    SyncCore -->|"BuildTree"| TreeBuilder
    SyncCore -->|"ManageObj"| ObjManager
    SyncCore -->|"ManagePeers"| PeerManager
    SyncCore -->|"HandleDelete"| DeletionManager
    TreeBuilder -->|"Diff"| LDiff
    TreeBuilder -->|"DiffOld"| OldDiff
    App -->|"CacheOps"| OCache
    App -->|"LogOps"| Logger
    DebugStat -->|"Stats"| Logger
    Utilities -->|"CryptoOps"| SecureService
    Utilities -->|"QueueOps"| SyncCore
    Utilities -->|"PeriodicSync"| SyncCore
    Utilities -->|"StorageUtil"| SpaceStorage

    %% Core to Storage
    SyncCore -->|"PersistGraph"| SpaceStorage
    BlockService -->|"PersistBlock"| FileBlockStore
    AccountService -->|"SaveNodeConf"| NodeConfStore
    SecureService -->|"LoadKeys"| KeyStorage

    %% P2P Sync Path
    SyncCore -->|"PeerSync"| SecureService
    Transport -->|"Stream"| PeerNode

    %% Legend
    subgraph Legend
        direction LR
        ext["External"]:::external
        net["Network & RPC"]:::network
        svc["Service & Protocol"]:::service
        corec["Core Engine & Utils"]:::core
        stor["Storage"]:::storage
    end

    %% Click Events
    click Transport "https://github.com/anyproto/any-sync/tree/main/net/transport"
    click StreamPool "https://github.com/anyproto/any-sync/tree/main/net/streampool"
    click ConnPool "https://github.com/anyproto/any-sync/tree/main/net/pool"
    click RPC "https://github.com/anyproto/any-sync/tree/main/net/rpc"
    click SecureService "https://github.com/anyproto/any-sync/tree/main/net/secureservice"
    click PeerService "https://github.com/anyproto/any-sync/tree/main/net/peerservice"
    click SpaceService "https://github.com/anyproto/any-sync/blob/main/commonspace/spaceservice.go"
    click FileService "https://github.com/anyproto/any-sync/blob/main/commonfile/fileservice/fileservice.go"
    click BlockService "https://github.com/anyproto/any-sync/blob/main/commonfile/fileservice/blockservice.go"
    click AccountService "https://github.com/anyproto/any-sync/blob/main/accountservice/accountservice.go"
    click ACLService "https://github.com/anyproto/any-sync/blob/main/acl/acl.go"
    click ConsensusClient "https://github.com/anyproto/any-sync/blob/main/consensus/consensusclient/client.go"
    click CoordinatorClient "https://github.com/anyproto/any-sync/blob/main/coordinator/coordinatorclient/coordinatorclient.go"
    click NameServiceClient "https://github.com/anyproto/any-sync/blob/main/nameservice/nameserviceclient/nameserviceclient.go"
    click IdentityRepo "https://github.com/anyproto/any-sync/blob/main/identityrepo/identityrepoproto/identityrepo.pb.go"
    click PaymentService "https://github.com/anyproto/any-sync/blob/main/paymentservice/paymentserviceclient/paymentserviceclient.go"
    click MetricService "https://github.com/anyproto/any-sync/blob/main/metric/metric.go"
    click App "https://github.com/anyproto/any-sync/blob/main/app/app.go"
    click Logger "https://github.com/anyproto/any-sync/blob/main/app/logger/log.go"
    click OCache "https://github.com/anyproto/any-sync/blob/main/app/ocache/ocache.go"
    click DebugStat "https://github.com/anyproto/any-sync/blob/main/app/debugstat/stat.go"
    click LDiff "https://github.com/anyproto/any-sync/blob/main/app/ldiff/diff.go"
    click OldDiff "https://github.com/anyproto/any-sync/blob/main/app/olddiff/diff.go"
    click ObjManager "https://github.com/anyproto/any-sync/blob/main/commonspace/objectmanager/objectmanager.go"
    click TreeBuilder "https://github.com/anyproto/any-sync/blob/main/commonspace/objecttreebuilder/treebuilder.go"
    click PeerManager "https://github.com/anyproto/any-sync/blob/main/commonspace/peermanager/peermanager.go"
    click DeletionManager "https://github.com/anyproto/any-sync/blob/main/commonspace/deletionmanager/deletionmanager.go"
    click SyncCore "https://github.com/anyproto/any-sync/blob/main/commonspace/sync/sync.go"
    click Utilities "https://github.com/anyproto/any-sync/tree/main/util"
    click SpaceStorage "https://github.com/anyproto/any-sync/blob/main/commonspace/spacestorage/spacestorage.go"
    click FileBlockStore "https://github.com/anyproto/any-sync/blob/main/commonfile/fileblockstore/fileblockstore.go"
    click NodeConfStore "https://github.com/anyproto/any-sync/blob/main/nodeconf/nodeconfstore/nodeconfstore.go"
    click KeyStorage "https://github.com/anyproto/any-sync/blob/main/util/crypto/keystorage.go"

    %% Styles
    classDef external fill:#CCCCCC,stroke:#000
    classDef network fill:#AEDFF7,stroke:#036
    classDef service fill:#C8E6C9,stroke:#2E7D32
    classDef core fill:#FFF59D,stroke:#F9A825
    classDef storage fill:#FFCC80,stroke:#EF6C00
```
