How netflix/conductor works
Main Components
• Core Engine: Represented by modules like “conductor-core”, “conductor-server”, and portions of “core” which implement the workflow execution, decider service, event processors, scheduling, and task mapping.
• API Layers: Separate components for REST (in “conductor-rest”) and gRPC (modules “conductor-grpc”, “conductor-grpc-server”, “conductor-grpc-client”) that expose endpoints for workflow control, task submission, and monitoring.
• Persistence: Multiple persistence modules exist:
– Redis based persistence (conductor-redis-persistence and redis-lock, redis-concurrency-limit) which handles workflow state and task queues.
– Cassandra and ES modules for alternate storage and indexing (e.g., “conductor-cassandra-persistence” and “conductor-es6-persistence”).
• External Task Providers/Workers: Modules like “conductor-http-task”, “json-jq-task”, and SDKs such as “conductor-client”, “conductor-client-spring”, “conductor-java-sdk” indicate that worker clients interact with the conductor engine to perform tasks.
• Client/UI: The “conductor-ui” module (with its Node/React components) provides a web-based dashboard. Also the “client” module and related SDKs support workflow creation and execution.
• Integrations/Extensions: Additional modules (like “awss3-storage”, “awssqs-event-queue”, “annotations”, “grpc-client”) show how the engine extends through external systems and custom annotations support.
Relationships & Interactions
• A Request Flow: Workflow definitions and task requests come in via REST or gRPC APIs to the server. The core engine then uses an internal decider service to “orchestrate” the workflow and direct tasks to appropriate workers.
• Persistence Operations: The engine continuously reads/writes workflow state to primary persistence (Redis) and indexes metadata in ElasticSearch. Alternative persistence modules (like Cassandra) may be used in different deployments.
• UI & Monitoring: The UI module interacts with the API layer to monitor live workflows. Additionally, logging, metric modules (inside core and client packages) feed data into external monitoring systems.
Architectural Patterns & Design Principles
• Modularity & Separation of Concerns: Each major feature (workflow orchestration, task execution, persistence, API exposure, UI) has its own module and can be represented separately.
• Extensibility: The support for community modules and plugins (e.g., alternative storage like AWS S3, SQS, gRPC client/server modules) should be clearly illustrated, showing hooks or extension points into the core engine.
• Use of Spring Boot: For server modules, the use of Spring Boot means that dependency injection, application configuration, and auto-configuration are key; showing a “Configuration & Bootstrapping” block can be useful.
Relevant Technologies
• Java (with JDK 17+), Spring Boot for the server.
• Redis, Cassandra, and Elasticsearch for persistence and indexing.
• Node.js and React (or similar) for the UI.
• gRPC & REST as the communication protocols.
• Docker for containerization (as indicated by extensive Docker files).
• Gradle as a build tool.