How netflix/genie works
Project type and purpose
• Project type: a federated Big Data orchestration & execution platform
• Purpose: expose a REST/gRPC service for job submission, dynamically select clusters and binaries, monitor execution via agents, record metadata, stream artifacts, and notify users
a. Clients
– genie-ui (JavaScript web UI)
– genie-client (Java REST client & CLI)
– genie-agent-app (CLI executor)
– Optional external clients (pygenie)
b. Service Tier
– genie-app (self-contained Spring Boot server)
– genie-web (core Spring libraries: REST controllers, HATEOAS, gRPC endpoints, service & scheduler tasks)
– genie-swagger (Swagger auto-config)
c. Agent Tier
– genie-agent (core agent library providing gRPC streaming, heartbeat, execution state machine)
d. Shared Libraries
– genie-common, genie-common-internal, genie-common-external (DTOs, converters, S3 integration, tracing, config)
– genie-proto (protobuf definitions for gRPC)
e. Persistence & Messaging
– Relational database (MySQL/PostgreSQL/H2 via Spring Data JPA)
– AWS S3 (optional file storage/backups)
– Amazon SNS (job notification bus)
f. Demo & Docs (genie-demo, genie-docs)
• UI → HTTP/HTTPS → genie-app (REST controllers)
• genie-client CLI → HTTP/HTTPS → genie-app (REST)
• genie-app → Swagger UI endpoint (via genie-swagger)
• genie-app → JPA → RDBMS (metadata, configs, audit)
• genie-app → SNS (job events)
• genie-app → S3 (archive artifacts via genie-common)
• genie-agent-app → gRPC → genie-app (streaming RPC endpoints defined in genie-proto)
• genie-agent-app → local filesystem/compute (invoke binaries, collect logs)
• genie-agent-app → fetch binaries/configs from server via gRPC file-stream
• genie-agent-app → heartbeat & kill RPCs → service
Architectural patterns
• Layered architecture in the service tier:
– Controller layer (Spring MVC REST & gRPC endpoints)
– Service layer (business logic, resource selection, script manager)
– Data layer (Spring Data JPA + Flyway/Liquibase migrations)
– Scheduler tasks (leader vs. node tasks via Spring Boot tasks)
– Observers & event bus (SNS publisher, tracing)
• Agent state machine: stages for handshake, download, execute, monitor, archive, cleanup
• Module extensibility: components auto-configure via Spring Boot (genie-web’s AutoConfiguration classes)