# nersc/interactem architecture

> Architecture diagram of the GitHub repository nersc/interactem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nersc/interactem
- Repository: https://github.com/nersc/interactem

## Overview

##### Main System Components

a. Frontend Layer:
- React-based web application (in frontend/interactEM)
- Client-side components for pipeline creation and visualization
- NATS client integration for real-time updates

b. Backend Services:
- App Service (backend/app) - Main API service
- Orchestrator Service (backend/orchestrator) - Pipeline management
- Agent Service (backend/agent) - Container deployment and management
- Operators Service (backend/operators) - Operator execution
- Metrics Service (backend/metrics) - System monitoring

c. Infrastructure Components:
- NATS messaging system (evident from nats-conf directory)
- Database (referenced in backend/app/alembic)
- Container runtime (referenced in Dockerfiles)
- HPC Integration Layer

**[Frontend Layer]**
- ↕
**[API Gateway/Backend Layer]**
- ↕
**[Orchestration Layer]**
- ↕
**[Execution Layer]**
- ↕
**[Infrastructure Layer]**

##### Key Relationships to Highlight

a. Data Flow Paths:
- Frontend → Backend API interactions
- NATS message routing between components
- Operator-to-operator data streaming
- Pipeline execution flow

b. Control Flow:
- Pipeline creation and management
- Operator deployment and lifecycle
- Container orchestration
- HPC resource allocation

##### Specific Implementation Details

- Blue rectangles: Core services (App, Orchestrator, Agent)
- Green hexagons: Operators (from operators/ directory)
- Yellow diamonds: Message brokers (NATS)
- Orange clouds: HPC resources

##### Component Interactions

Following key interactions:
- Frontend → Backend API (REST/WebSocket)
- Services → NATS (pub/sub)
- Operator → Operator (ZMQ, evident from operators/messengers/zmq.py)
- Agent → Container Runtime
- Orchestrator → HPC Resources

##### Deployment view showing HPC integration

Remember to emphasize the distributed nature of the system and how it handles both edge and HPC resources while maintaining the interactive nature of the frontend experience.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef frontend fill:#a8d1ff,stroke:#333
    classDef backend fill:#afd7af,stroke:#333
    classDef infrastructure fill:#ffb366,stroke:#333
    classDef database fill:#c2a5cf,stroke:#333
    classDef messaging fill:#ffed99,stroke:#333
    classDef security fill:#ff9999,stroke:#333
    classDef operator fill:#98fb98,stroke:#333

    %% Frontend Layer
    subgraph Frontend
        FE["React Frontend"]:::frontend
        PipeViz["Pipeline Visualization"]:::frontend
        OpComp["Operator Components"]:::frontend
        NatsClient["NATS Client"]:::frontend
    end
    click FE "https://github.com/nersc/interactem/tree/main/frontend/interactEM/"
    click PipeViz "https://github.com/nersc/interactem/blob/main/frontend/interactEM/src/components/pipelineflow.tsx"
    click OpComp "https://github.com/nersc/interactem/blob/main/frontend/interactEM/src/components/operatornode.tsx"
    click NatsClient "https://github.com/nersc/interactem/blob/main/frontend/interactEM/src/nats/NatsContext.tsx"

    %% Backend Layer
    subgraph Backend
        AppService["App Service"]:::backend
        Security["Security Layer"]:::security
        APIRoutes["API Routes"]:::backend
        DBModels["Database Models"]:::backend
    end
    click AppService "https://github.com/nersc/interactem/tree/main/backend/app/"
    click Security "https://github.com/nersc/interactem/blob/main/backend/app/app/core/security.py"
    click APIRoutes "https://github.com/nersc/interactem/tree/main/backend/app/app/api/routes/"
    click DBModels "https://github.com/nersc/interactem/blob/main/backend/app/app/models.py"

    %% Orchestration Layer
    subgraph Orchestration
        Orchestrator["Orchestrator Service"]:::backend
        Agent["Agent Service"]:::backend
        OpService["Operators Service"]:::backend
        Metrics["Metrics Service"]:::backend
    end
    click Orchestrator "https://github.com/nersc/interactem/blob/main/backend/orchestrator/orchestrator/orchestrator.py"
    click Agent "https://github.com/nersc/interactem/blob/main/backend/agent/agent/agent.py"
    click OpService "https://github.com/nersc/interactem/blob/main/backend/operators/operators/operator.py"
    click Metrics "https://github.com/nersc/interactem/blob/main/backend/metrics/metrics/metrics.py"

    %% Pipeline Management
    subgraph PipelineManagement
        PipeModel["Pipeline Models"]:::backend
        PipeLogic["Pipeline Logic"]:::backend
        PipeAPI["Pipeline API"]:::backend
    end
    click PipeModel "https://github.com/nersc/interactem/blob/main/backend/core/core/models/pipeline.py"
    click PipeLogic "https://github.com/nersc/interactem/blob/main/backend/core/core/pipeline.py"
    click PipeAPI "https://github.com/nersc/interactem/blob/main/backend/app/app/api/routes/pipelines.py"

    %% Infrastructure Layer
    subgraph Infrastructure
        NATS[("NATS Messaging")]:::messaging
        DB[("Database")]:::database
        K8s["Kubernetes"]:::infrastructure
        HPC["HPC Resources"]:::infrastructure
    end
    click NATS "https://github.com/nersc/interactem/tree/main/nats-conf/"
    click DB "https://github.com/nersc/interactem/tree/main/backend/app/app/alembic/"
    click K8s "https://github.com/nersc/interactem/tree/main/helm/"

    %% Operator Layer
    subgraph Operators
        OpBase["Operator Base"]:::operator
        ZMQ["ZMQ Messenger"]:::messaging
        IndivOps["Individual Operators"]:::operator
    end
    click OpBase "https://github.com/nersc/interactem/blob/main/backend/operators/operators/operator.py"
    click ZMQ "https://github.com/nersc/interactem/blob/main/backend/operators/operators/messengers/zmq.py"
    click IndivOps "https://github.com/nersc/interactem/tree/main/operators/"

    %% Relationships
    FE --> PipeViz
    FE --> OpComp
    FE --> NatsClient
    NatsClient --> NATS
    
    AppService --> Security
    AppService --> APIRoutes
    AppService --> DBModels
    
    Orchestrator --> NATS
    Agent --> NATS
    OpService --> NATS
    Metrics --> NATS
    
    PipeModel --> DBModels
    PipeLogic --> PipeModel
    PipeAPI --> PipeLogic
    
    OpBase --> ZMQ
    OpBase --> IndivOps
    
    APIRoutes --> DB
    Orchestrator --> K8s
    K8s --> HPC

    %% Legend
    subgraph Legend
        L1["Frontend Components"]:::frontend
        L2["Backend Services"]:::backend
        L3["Infrastructure"]:::infrastructure
        L4["Database"]:::database
        L5["Messaging"]:::messaging
        L6["Security"]:::security
        L7["Operators"]:::operator
    end
```
