How rafaelvp-db/databricks-end-to-end-streaming works
Main Components
• Kafka Brokers: Represent both AWS MSK and Confluent Kafka as two external streaming platforms. Indicate that each has its own Schema Registry integration (AWS Glue Schema Registry for MSK and Confluent Schema Registry for Confluent).
• Databricks Environment: Show that Databricks is at the center of the streaming and batch processing. There are several notebooks grouped by functionality:
– Ingestion/Producer Notebooks: separate notebooks under “notebooks/confluent” and “notebooks/msk” act as random event generators (producers) that publish to Kafka.
– Medallion Processing Notebooks: The “notebooks/medallion” directory contains the bronze, silver, and gold notebooks representing different stages of processing.
• Avro Schemas: Indicate a component for Avro schema definitions (the “schema” folder) that feed into the Schema Registries.
• Job Orchestration & Deployment: Include the Databricks Jobs (files in “jobs” folder) and the Terraform Infrastructure-as-Code (IaC) scripts (in “terraform” folder) which deploy and configure these pipelines.
Architectural Patterns and Key Design Principles
• Medallion Architecture: Specifically emphasize the bronze, silver, and gold stages to clarify how data evolves through the pipeline.
• Integration Specifics: Annotate differences in libraries and dependencies (for example, the specific Maven dependencies and jar versions required for Confluent versus MSK) to hint why separate clusters might be needed.
Including Relevant Technologies and Frameworks
• Databricks (Spark Structured Streaming and Delta Lake as implied by the medallion notebooks).
• Apache Kafka, with two variants: AWS MSK and Confluent Kafka.
• Avro schema management and Schema Registry integration (via AWS Glue and Confluent’s schema registry).
• Terraform for IaC deployment.
• The various language integrations (Python and Scala) in the producer and ingestion pipelines.
• That all infrastructure components are accounted for, such as the dependency on proper authentication (e.g., secret scopes in Databricks) and the need for specific configurations to support Schema Registry integration for both Kafka platforms.
• Don’t overcomplicate matters—start by segmenting the system into as many logical components as possible in your initial draft, then refine connections based on workflow dependencies.