How nersc/interactem works
Main System Components
a. Frontend Layer:
• React-based web application (in frontend/interactEM)
• Client-side components for pipeline creation and visualization
• NATS client integration for real-time updates
b. Backend Services:
• App Service (backend/app) - Main API service
• Orchestrator Service (backend/orchestrator) - Pipeline management
• Agent Service (backend/agent) - Container deployment and management
• Operators Service (backend/operators) - Operator execution
• Metrics Service (backend/metrics) - System monitoring
c. Infrastructure Components:
• NATS messaging system (evident from nats-conf directory)
• Database (referenced in backend/app/alembic)
• Container runtime (referenced in Dockerfiles)
• HPC Integration Layer
[Frontend Layer]
• ↕
[API Gateway/Backend Layer]
• ↕
[Orchestration Layer]
• ↕
[Execution Layer]
• ↕
[Infrastructure Layer]
Key Relationships to Highlight
a. Data Flow Paths:
• Frontend → Backend API interactions
• NATS message routing between components
• Operator-to-operator data streaming
• Pipeline execution flow
b. Control Flow:
• Pipeline creation and management
• Operator deployment and lifecycle
• Container orchestration
• HPC resource allocation
Specific Implementation Details
• Blue rectangles: Core services (App, Orchestrator, Agent)
• Green hexagons: Operators (from operators/ directory)
• Yellow diamonds: Message brokers (NATS)
• Orange clouds: HPC resources
Component Interactions
Following key interactions:
• Frontend → Backend API (REST/WebSocket)
• Services → NATS (pub/sub)
• Operator → Operator (ZMQ, evident from operators/messengers/zmq.py)
• Agent → Container Runtime
• Orchestrator → HPC Resources
Deployment view showing HPC integration
Remember to emphasize the distributed nature of the system and how it handles both edge and HPC resources while maintaining the interactive nature of the frontend experience.