How cylc/cylc-flow works
Cylc Flow is a general-purpose workflow engine with explicit support for cycling workflows, branching, HPC execution, and runtime monitoring and control. The principal path is workflow authoring and validation, installation and launch, dependency-aware scheduling, task dispatch to execution platforms, and persisted/observable workflow state. The graph includes the CLI, scheduler and task runtime, configuration and cycling, job execution, state, and communications/UI branches. README and tree establish product capabilities; sampled excerpts specifically support graph parsing, network client/server behavior, and the TUI. Other internal wiring is shown only where the product structure supports it.
Main entry points: scheduler.py Scheduler · config.py Config loading · task_pool.py Task pool · client.py Runtime API client
Components 24
Workflow CLI
- Validate command
cylc/flow/scripts/validate.py - Install command
cylc/flow/scripts/install.py - Play command
cylc/flow/scripts/play.py
Workflow Definition
- Workflow files
cylc/flow/workflow_files.py - Config loading
cylc/flow/config.py - Workflow config spec
cylc/flow/cfgspec/workflow.py - Config validation
cylc/flow/parsec/validate.py - Dependency graph parser
cylc/flow/graph_parser.py - Cycle arithmetic
cylc/flow/cycling/loader.py
Scheduler Runtime
- Scheduler
cylc/flow/scheduler.py - Task pool
cylc/flow/task_pool.py - Task state and proxy
cylc/flow/task_proxy.py - External triggers
cylc/flow/xtrigger_mgr.py
Task Execution
- Job manager
cylc/flow/task_job_mgr.py - Platform job runners
cylc/flow/job_runner_mgr.py
State and Control
- Workflow database
cylc/flow/workflow_db_mgr.py - Workflow data store
cylc/flow/data_store_mgr.py - Runtime commands
cylc/flow/scripts/trigger.py
Communications and UI
- Scheduler API server
cylc/flow/network/server.py - Runtime API client
cylc/flow/network/client.py - Terminal interface
cylc/flow/tui/app.py
Other components
- Workflow user
- Execution platform
- Web UI service
Connections 28
- Workflow user → Validate command: validates
- Workflow user → Install command: installs
- Workflow user → Play command: starts
- Validate command → Config loading: loads config
- Install command → Workflow files: installs workflow
- Play command → Scheduler: starts scheduler
- Config loading → Workflow files: reads definition
- Config loading → Workflow config spec: uses schema
- Workflow config spec → Config validation: validates values
- Config loading → Dependency graph parser: parses dependencies
- Config loading → Cycle arithmetic: resolves cycling
- Scheduler → Config loading: loads workflow
- Scheduler → Task pool: schedules tasks
- Scheduler → Workflow database: persists state
- Scheduler → Workflow data store: updates workflow data
- Scheduler → Scheduler API server: hosts API
- Task pool → Task state and proxy: manages tasks
- Task pool → External triggers: checks triggers
- Task pool → Job manager: dispatches jobs
- Job manager → Platform job runners: selects runner
- Platform job runners → Execution platform: submits jobs
- Execution platform → Job manager: reports job results
- Runtime commands → Runtime API client: issues requests
- Runtime API client → Scheduler API server: sends requests
- Scheduler API server → Workflow data store: reads workflow data
- Workflow user → Terminal interface: monitors workflow
- Terminal interface → Runtime API client: requests updates
- Web UI service → Runtime API client: connects optionally