# rafaelvp-db/databricks-end-to-end-streaming architecture

> Architecture diagram of the GitHub repository rafaelvp-db/databricks-end-to-end-streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rafaelvp-db/databricks-end-to-end-streaming
- Repository: https://github.com/rafaelvp-db/databricks-end-to-end-streaming

## Overview

##### Main Components
- Kafka Brokers: Represent both AWS MSK and Confluent Kafka as two external streaming platforms. Indicate that each has its own Schema Registry integration (AWS Glue Schema Registry for MSK and Confluent Schema Registry for Confluent).
- Databricks Environment: Show that Databricks is at the center of the streaming and batch processing. There are several notebooks grouped by functionality:
– Ingestion/Producer Notebooks: separate notebooks under “notebooks/confluent” and “notebooks/msk” act as random event generators (producers) that publish to Kafka.
– Medallion Processing Notebooks: The “notebooks/medallion” directory contains the bronze, silver, and gold notebooks representing different stages of processing.
- Avro Schemas: Indicate a component for Avro schema definitions (the “schema” folder) that feed into the Schema Registries.
- Job Orchestration & Deployment: Include the Databricks Jobs (files in “jobs” folder) and the Terraform Infrastructure-as-Code (IaC) scripts (in “terraform” folder) which deploy and configure these pipelines.

##### Architectural Patterns and Key Design Principles
- Medallion Architecture: Specifically emphasize the bronze, silver, and gold stages to clarify how data evolves through the pipeline.
- Integration Specifics: Annotate differences in libraries and dependencies (for example, the specific Maven dependencies and jar versions required for Confluent versus MSK) to hint why separate clusters might be needed.

##### Including Relevant Technologies and Frameworks
- Databricks (Spark Structured Streaming and Delta Lake as implied by the medallion notebooks).
- Apache Kafka, with two variants: AWS MSK and Confluent Kafka.
- Avro schema management and Schema Registry integration (via AWS Glue and Confluent’s schema registry).
- Terraform for IaC deployment.
- The various language integrations (Python and Scala) in the producer and ingestion pipelines.

- That all infrastructure components are accounted for, such as the dependency on proper authentication (e.g., secret scopes in Databricks) and the need for specific configurations to support Schema Registry integration for both Kafka platforms.
- Don’t overcomplicate matters—start by segmenting the system into as many logical components as possible in your initial draft, then refine connections based on workflow dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
      AWS_Broker["AWS MSK Broker"]:::external
      AWS_Schema["AWS Glue Schema Registry"]:::external
      Confluent_Broker["Confluent Kafka Broker"]:::external
      Confluent_Schema["Confluent Schema Registry"]:::external
    end

    %% Databricks Environment
    subgraph "Databricks Environment"
      subgraph "Producer Notebooks"
        Confluent_Prod_Pyt["Databricks Producer Notebook (Python)"]:::producer
        Confluent_Prod_Sc["Databricks Producer Notebook (Scala)"]:::producer
        MSK_Prod_Sc["Databricks Producer Notebook (MSK - Scala)"]:::producer
      end
      subgraph "Ingestion Notebooks"
        MSK_Ingest["Databricks Ingestion Notebook (MSK)"]:::ingestion
      end
      subgraph "Medallion Processing"
        Bronze["Bronze Notebook"]:::medallion
        Silver["Silver Notebook"]:::medallion
        Gold["Gold Notebook"]:::medallion
      end
    end

    %% Supporting Components
    subgraph "Avro Schema Definitions"
      Avro_v1["Avro Schema v1"]:::avro
      Avro_v2["Avro Schema v2"]:::avro
    end

    %% Infrastructure & Deployment
    subgraph "Job Configurations"
      Jobs_Confluent["Databricks Jobs (Confluent)"]:::infrastructure
      Jobs_MSK["Databricks Jobs (MSK)"]:::infrastructure
    end

    subgraph "Terraform IaC Scripts"
      Terraform_Confluent["Terraform: confluent_ingestion_job.tf"]:::infrastructure
      Terraform_MSK["Terraform: msk_ingestion_job.tf"]:::infrastructure
      Terraform_Vars["Terraform: variables.tf"]:::infrastructure
    end

    %% Connections: Producers -> Brokers
    Confluent_Prod_Pyt -->|"publishEvent"| Confluent_Broker
    Confluent_Prod_Sc -->|"publishEvent"| Confluent_Broker
    MSK_Prod_Sc -->|"publishEvent"| AWS_Broker

    %% Avro Schemas -> Schema Registries
    Avro_v1 -->|"ProvidesSchema"| AWS_Schema
    Avro_v1 -->|"ProvidesSchema"| Confluent_Schema
    Avro_v2 -->|"ProvidesSchema"| AWS_Schema
    Avro_v2 -->|"ProvidesSchema"| Confluent_Schema

    %% Brokers -> Ingestion
    AWS_Broker -->|"streamData"| MSK_Ingest
    Confluent_Broker -->|"streamData"| MSK_Ingest

    %% Ingestion -> Medallion Processing
    MSK_Ingest -->|"RawData"| Bronze
    Bronze -->|"RefinedData"| Silver
    Silver -->|"TransformedData"| Gold

    %% Infrastructure interactions
    Jobs_Confluent -->|"TriggerJob"| MSK_Ingest
    Jobs_MSK -->|"TriggerJob"| MSK_Ingest
    Terraform_Confluent -->|"DeployJob"| Jobs_Confluent
    Terraform_MSK -->|"DeployJob"| Jobs_MSK
    Terraform_Vars -->|"Configure"| Jobs_Confluent
    Terraform_Vars -->|"Configure"| Jobs_MSK

    %% Styles
    classDef external fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef producer fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef ingestion fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef medallion fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef avro fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef infrastructure fill:#FFB6C1,stroke:#000,stroke-width:2px;

    %% Click Events
    click Confluent_Prod_Pyt "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/confluent/python/producer.py"
    click Confluent_Prod_Sc "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/confluent/scala/ingest_raw.scala"
    click MSK_Prod_Sc "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/msk/producer.scala"
    click MSK_Ingest "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/msk/ingest.scala"
    click Bronze "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/medallion/bronze.py"
    click Silver "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/medallion/silver.py"
    click Gold "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/notebooks/medallion/gold.py"
    click Avro_v1 "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/schema/product-v1.avsc"
    click Avro_v2 "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/schema/product-v2.avsc"
    click Jobs_Confluent "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/jobs/confluent.json"
    click Jobs_MSK "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/jobs/msk.json"
    click Terraform_Confluent "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/terraform/confluent_ingestion_job.tf"
    click Terraform_MSK "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/terraform/msk_ingestion_job.tf"
    click Terraform_Vars "https://github.com/rafaelvp-db/databricks-end-to-end-streaming/blob/main/terraform/variables.tf"
```
