# tf-encrypted/moose architecture

> Architecture diagram of the GitHub repository tf-encrypted/moose, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tf-encrypted/moose
- Repository: https://github.com/tf-encrypted/moose

## Overview

##### Main Components

a. Core Framework Components:
- Compiler (moose/src/compilation/)
- Runtime Engine (moose/src/execution/)
- Python eDSL & Bindings (pymoose/)
- Networking Layer (moose/src/networking/)

b. Execution Components:
- Host Placements
- Replicated Secret Sharing Placement
- Virtual Execution Units

c. Binary Components:
- Comet (Network Node)
- Elk (Compiler)
- Rudolph (Runtime)
- Vixen (Orchestration)

##### Component Relationships

**[Python eDSL] → [Compiler (Elk)] → [Computation Graph]**
- ↓
**[Storage Layer] ← [Runtime (Rudolph)] ← [Optimized Graph]**
- ↕                    ↕
**[Network Layer (Comet)] ↔ [MPC Protocol Layer]**

##### Detailed Component Breakdown

a. Compiler Layer:
- Parsing flow
- Type checking
- Graph optimization
- Code generation

b. Runtime Layer:
- Execution context
- Kernel management
- Synchronization
- Memory management

c. Security Layer:
- Replicated secret sharing
- Secure computation protocols
- Encryption/decryption

##### Important Elements to Highlight

- Data flow directions between components
- Security boundaries
- Protocol transitions
- Storage interfaces
- API endpoints

##### Implementation Details

Technical annotations for:
- Protocol specifications (gRPC)
- Supported data types
- Security protocols
- Storage formats (NumPy, CSV)

##### Deployment View

How components are distributed:
- Multiple Comet instances
- Network topology
- Data source connections
- Client connections

##### Extensibility points

Keep the abstraction level high enough to be understandable but detailed enough to show the security and distributed computing aspects that make Moose unique.

This system design should reflect Moose's architecture as a secure distributed dataflow framework with emphasis on its MPC capabilities and machine learning support.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        L1[Core Components]:::core
        L2[Data Processing]:::data
        L3[Security Components]:::security
        L4[Network Components]:::network
        L5[Storage Components]:::storage
    end

    subgraph "Frontend Layer"
        PythonEDSL["Python eDSL"]:::core
        MLComponents["ML Components"]:::data
    end

    subgraph "Compilation Layer"
        Compiler["Compiler (Elk)"]:::core
        Parser["Parser"]:::core
        TypeChecker["Type Checker"]:::core
        Optimizer["Graph Optimizer"]:::core
        CodeGen["Code Generator"]:::core
    end

    subgraph "Runtime Layer"
        Runtime["Runtime (Rudolph)"]:::core
        ExecContext["Execution Context"]:::core
        KernelMgr["Kernel Manager"]:::core
        SyncMgr["Sync Manager"]:::core
        MemoryMgr["Memory Manager"]:::core
    end

    subgraph "Security Layer"
        SecureComp["Secure Computation"]:::security
        RSS["Replicated Secret Sharing"]:::security
        Encryption["Encryption/Decryption"]:::security
    end

    subgraph "Network Layer"
        NetworkNode["Network Node (Comet)"]:::network
        Orchestrator["Orchestrator (Vixen)"]:::network
        Choreography["Choreography System"]:::network
        Protocols["Protocol Definitions"]:::network
    end

    subgraph "Storage Layer"
        Storage["Storage System"]:::storage
        DataFormats["Data Formats\n(NumPy, CSV)"]:::storage
    end

    PythonEDSL --> Compiler
    MLComponents --> Compiler
    
    Compiler --> Parser
    Parser --> TypeChecker
    TypeChecker --> Optimizer
    Optimizer --> CodeGen
    
    CodeGen --> Runtime
    Runtime --> ExecContext
    Runtime --> KernelMgr
    Runtime --> SyncMgr
    Runtime --> MemoryMgr
    
    ExecContext --> SecureComp
    SecureComp --> RSS
    SecureComp --> Encryption
    
    Runtime --> NetworkNode
    NetworkNode --> Orchestrator
    NetworkNode --> Choreography
    Choreography --> Protocols
    
    Runtime --> Storage
    Storage --> DataFormats

    classDef core fill:#3498db
    classDef data fill:#2ecc71
    classDef security fill:#e67e22
    classDef network fill:#9b59b6
    classDef storage fill:#95a5a6

    click PythonEDSL "https://github.com/tf-encrypted/moose/tree/main/pymoose/pymoose/edsl/"
    click MLComponents "https://github.com/tf-encrypted/moose/tree/main/pymoose/pymoose/predictors/"
    click Compiler "https://github.com/tf-encrypted/moose/tree/main/moose/src/compilation/"
    click Runtime "https://github.com/tf-encrypted/moose/tree/main/moose/src/execution/"
    click NetworkNode "https://github.com/tf-encrypted/moose/tree/main/moose/src/networking/"
    click Orchestrator "https://github.com/tf-encrypted/moose/tree/main/moose/src/bin/vixen/"
    click Storage "https://github.com/tf-encrypted/moose/tree/main/moose/src/storage/"
    click Choreography "https://github.com/tf-encrypted/moose/tree/main/moose/src/choreography/"
    click SecureComp "https://github.com/tf-encrypted/moose/tree/main/moose/src/replicated/"
    click Protocols "https://github.com/tf-encrypted/moose/blob/main/moose/protos/networking.proto"
```
