How windmill-labs/windmill-benchmarks works
This repository is a benchmark suite: it compares workflow orchestrators and separately benchmarks TPC-H data integrations. The main user workflow is to run platform-specific benchmarks, then inspect or aggregate results; the pipeline branch runs analytical queries over datasets and writes outputs. Sampled code confirms some Hatchet and Temporal wiring and Airflow’s S3 data flow. Other platform implementations are represented by their tree paths, but their internal wiring is not inferred from filenames.
Main entry points: run_bench.py Hatchet runner · main.go Temporal starter · worker.py Hatchet worker · worker.go Temporal worker
Components 22
Orchestrator comparisons
- Windmill benchmark
competitors/2026-09/windmill/wm_bench.py - Windmill dedicated
competitors/2026-09/windmill/wm_dedicated_bench.py - Airflow benchmark
competitors/2026-09/airflow/airflow_harness.py - Hatchet runner
competitors/2026-09/hatchet/app/run_bench.py - Hatchet definitions
competitors/2026-09/hatchet/app/bench_defs.py - Hatchet worker
competitors/2026-09/hatchet/app/worker.py - Kestra benchmark
competitors/2026-09/kestra/kestra_bench.py - Prefect benchmark
competitors/2026-09/prefect/run_bench.py - Temporal starter
competitors/temporal/flows/go/cmd/starter/main.go - Temporal worker
competitors/temporal/flows/go/worker/worker.go - Temporal TypeScript
competitors/temporal/flows/typescript/bench_10/src/worker.ts
Results analysis
- Result aggregation
competitors/2026-09/aggregate.py - Result validation
competitors/2026-09/validate.py - Competitor analysis
competitors/hatchet/benchmark_analyzer.py
Data integrations
- Airflow TPC-H
pipelines/airflow/tpc_h.py - DuckDB TPC-H
pipelines/duckdb/tpc_h.py - Polars TPC-H
pipelines/polars/tpc_h.py - Spark TPC-H
pipelines/spark/tpc_h.py
Other components
- Benchmark user
- Orchestrator services
- Pipeline user
- S3 storage
Connections 23
- Benchmark user → Windmill benchmark: runs
- Benchmark user → Windmill dedicated: runs
- Benchmark user → Airflow benchmark: runs
- Benchmark user → Hatchet runner: runs
- Benchmark user → Kestra benchmark: runs
- Benchmark user → Prefect benchmark: runs
- Benchmark user → Temporal starter: runs
- Hatchet runner → Hatchet definitions: invokes
- Hatchet worker → Hatchet definitions: registers
- Hatchet runner → Orchestrator services: submits runs
- Hatchet worker → Orchestrator services: polls tasks
- Temporal starter → Temporal worker: launches process
- Temporal starter → Orchestrator services: starts workflow
- Temporal worker → Orchestrator services: polls tasks
- Temporal TypeScript → Orchestrator services: starts workflow
- Benchmark user → Result aggregation: aggregates results
- Benchmark user → Result validation: validates results
- Benchmark user → Competitor analysis: analyzes results
- Pipeline user → Airflow TPC-H: runs queries
- Pipeline user → DuckDB TPC-H: runs queries
- Pipeline user → Polars TPC-H: runs queries
- Pipeline user → Spark TPC-H: runs queries
- Airflow TPC-H → S3 storage: reads and writes