How openchemistry/distiller works
Main System Components
a) Client Layer:
• React Frontend (under /frontend/distiller)
• Web Interface for monitoring and control
• Real-time data visualization components
b) API Layer:
• FastAPI Backend (under /backend/app)
• REST endpoints for various services
• Authentication and authorization
c) Processing Layer:
• Faust Workers (under /backend/faust)
• Multiple specialized workers (job, notebook, scan, etc.)
• Template processing
d) Monitoring Layer:
• Watch CLI (under /cli/watch)
• Custodian service (under /cli/custodian)
• Real-time file monitoring
e) External Services:
• NERSC HPC
• Jupyter Hub
• Kafka Message Broker
• Database (SQL, based on alembic migrations)
Component Relationships
Frontend ←→ FastAPI Backend ←→ Kafka ←→ Faust Workers ←→ NERSC HPC
• ↑ ↑
• ↓ ↓
• Database File Monitoring System
Specific Implementation Details
a) Frontend Architecture:
• React components (/frontend/distiller/src/components)
• Feature modules (/frontend/distiller/src/features)
• State management (Redux, based on store.ts)
b) Backend Services:
• API endpoints (/backend/app/api/api_v1/endpoints)
• Authentication flow
• Database models (/backend/app/models)
• Schema definitions (/backend/app/schemas)
c) Worker Architecture:
• Job processing pipeline
• Notebook generation
• Scan processing
• File monitoring system
Integration Points
Integration with:
• NERSC Superfacility API
• Jupyter Hub
• Stempy processing library
• Microscopy equipment