How nextflow-io/nextflow works
Nextflow turns a user-authored workflow and configuration into a compiled dataflow program, schedules process tasks, manages reproducibility, and materializes outputs. The sampled code strongly supports configuration parsing, AST construction, task hashing, file handling, and S3 transfer control; broader runtime and platform capabilities are represented from the README and directory structure, with unsampled links kept at subsystem boundaries. Optional cloud and scheduler integrations remain separate from the core workflow path.
Main entry points: nextflow Task executor · ScriptCompiler.java Script compiler · ConfigParser.java Config parser · ScriptParser.java Script parser
Components 29
Language frontend
- Script compiler
modules/nextflow/src/main/groovy/nextflow/script/parser/v2/ScriptCompiler.java - Script parser
modules/nf-lang/src/main/java/nextflow/script/control/ScriptParser.java - Script AST
modules/nf-lang/src/main/java/nextflow/script/ast/ScriptNode.java - Config parser
modules/nf-lang/src/main/java/nextflow/config/control/ConfigParser.java - Config AST
modules/nf-lang/src/main/java/nextflow/config/parser/ConfigParserPlugin.java - Config resolver
modules/nf-lang/src/main/java/nextflow/config/control/ConfigResolveVisitor.java
Workflow runtime
- Workflow runtime
modules/nextflow - Workflow DSL
modules/nf-lang/src/main/java/nextflow/script/dsl/WorkflowDsl.java - Dataflow channels
modules/nf-lang/src/main/java/nextflow/script/types/Channel.java - Task closure
modules/nextflow/src/main/groovy/nextflow/script/TaskClosure.java - Task executor
modules/nextflow
Reproducibility
- Task hashing
modules/nf-commons/src/main/nextflow/util/HashBuilder.java - Task cache
modules/nf-commons/src/main/nextflow/util/CacheHelper.java - File outputs
modules/nf-commons/src/main/nextflow/file/CopyMoveHelper.java - Software environments
modules/nextflow
Execution integrations
- Local execution
modules/nextflow - AWS execution
plugins/nf-amazon - Azure execution
plugins/nf-azure - Google execution
plugins/nf-google - Kubernetes execution
plugins/nf-k8s - S3 storage
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Other components
- Workflow author
- Local machine
- HPC scheduler
- AWS Batch
- Azure Batch
- Google Cloud Batch
- Kubernetes cluster
- Amazon S3
Connections 30
- Workflow author → Script compiler: provides script
- Workflow author → Config parser: provides config
- Script compiler → Script parser: parses script
- Script parser → Script AST: builds AST
- Script compiler → Workflow runtime: starts workflow
- Config parser → Config AST: builds AST
- Config parser → Config resolver: resolves config
- Config resolver → Workflow runtime: supplies settings
- Workflow runtime → Workflow DSL: interprets DSL
- Workflow runtime → Dataflow channels: creates channels
- Workflow runtime → Task closure: dispatches tasks
- Workflow runtime → Task executor: schedules tasks
- Task executor → Task hashing: computes hashes
- Task executor → Task cache: reads cache
- Task executor → Software environments: prepares environment
- Task executor → File outputs: writes outputs
- Task executor → Local execution: runs locally
- Task executor → AWS execution: dispatches tasks
- Task executor → Azure execution: dispatches tasks
- Task executor → Google execution: dispatches tasks
- Task executor → Kubernetes execution: dispatches tasks
- Local execution → Local machine: runs on
- Local execution → HPC scheduler: submits to
- AWS execution → AWS Batch: submits to
- Azure execution → Azure Batch: submits to
- Google execution → Google Cloud Batch: submits to
- Kubernetes execution → Kubernetes cluster: submits to
- Task executor → S3 storage: transfers files
- S3 storage → Amazon S3: reads/writes
- File outputs → Workflow author: returns results