# segmentio/terraform-segment-data-lakes architecture

> Architecture diagram of the GitHub repository segmentio/terraform-segment-data-lakes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/segmentio/terraform-segment-data-lakes
- Repository: https://github.com/segmentio/terraform-segment-data-lakes

## Overview

1. **What this repo is**
   - This is an infrastructure repository, not an application codebase.
   - It contains Terraform modules for provisioning a Segment data lake in **two cloud boundaries**: **AWS** and **Azure**.
   - The repo is organized as reusable modules plus example/test fixtures, so the architecture is “module composition” rather than a single monolithic stack.

2. **Top-level split: AWS vs Azure**
   - The repository is cleanly divided into two independent roots:
     - `aws_datalake/`
     - `azure_datalake/`
   - Each root has its own module set, README, and test fixture, which suggests the two clouds are deployed and validated separately.

3. **AWS subsystem**
   - AWS is broken into four Terraform modules:
     - `iam` for identities, permissions, and cross-account access
     - `lakeformation` for data lake governance and permissions
     - `glue` for data catalog / ETL-oriented metadata services
     - `emr` for managed compute / processing
   - These are the core AWS building blocks for a governed lake: access control, catalog/governance, metadata/transform services, and distributed processing.

4. **AWS data/control flow**
   - The likely dependency order is:
     - **IAM** establishes roles/policies and external trust relationships first.
     - **Lake Formation** then uses those permissions to control access to data assets.
     - **Glue** provides cataloging and metadata services that Lake Formation and consumers rely on.
     - **EMR** consumes the IAM and governance setup to run Spark/Hadoop-style processing against the lake.
   - In architecture terms, IAM is the foundational security boundary; the other modules sit on top of it.

5. **Azure subsystem**
   - Azure is also split into four modules:
     - `StorageAccount` for the lake storage layer
     - `serviceprincipal` for identity and access
     - `databricks` for the analytics/compute workspace
     - `mysql` for a supporting relational service
   - This is a different shape from AWS: Azure centers on storage + identity + Databricks compute, with MySQL as an adjacent dependency.

6. **Azure data/control flow**
   - The typical sequence is:
     - **StorageAccount** creates the durable data lake storage boundary.
     - **serviceprincipal** creates the identity used for automation and access.
     - **databricks** integrates compute/workspace resources with storage and identity.
     - **mysql** provisions a relational dependency that appears to support the wider platform rather than the lake itself.
   - The key architectural boundary here is the separation between storage, identity, and analytics workspace.

7. **Module design and boundaries**
   - The repo favors small, purpose-built Terraform modules over a single environment stack.
   - Each module has its own `main.tf`, `variables.tf`, and `output.tf`, which indicates:
     - inputs are explicit
     - outputs are used to wire modules together
     - stateful concerns are separated per component
   - This makes the architecture diagram naturally component-based: boxes for modules, arrows for outputs/inputs.

8. **Testing structure**
   - Both cloud roots have a `test/` directory with a `test_fixture/` and integration test files.
   - The AWS side includes a Ruby test file under `test/integration/default/`, and the repo has `Gemfile`, `Gemfile.lock`, and `.kitchen.yml`, which strongly suggests **Test Kitchen / Ruby-based Terraform integration testing**.
   - The fixtures act as minimal composed environments to validate that modules can be wired together and applied successfully.

9. **CI and delivery**
   - `.buildkite/pipeline.yml` shows this repo is validated in Buildkite CI.
   - The architecture of delivery is therefore:
     - commit Terraform/module changes
     - CI runs validation/integration tests
     - modules are consumed by downstream environment stacks
   - This repo does not appear to deploy a live runtime by itself; it produces reusable infrastructure primitives.

10. **Technology and runtime boundaries**
   - The primary implementation language is **Terraform (HCL)**.
   - Supporting tooling is **Ruby** for test orchestration and **Buildkite** for CI.
   - The repo targets **Terraform 0.12+** and explicitly notes AWS provider version constraints to avoid breaking changes, which is important for dependency compatibility in the architecture.

11. **External/cloud service boundaries**
   - AWS side materially depends on:
     - IAM
     - Lake Formation
     - Glue
     - EMR
   - Azure side materially depends on:
     - Storage Account
     - Databricks
     - Service Principal / Azure AD identity
     - MySQL
   - These are the real external system boundaries the diagram should show, not internal code packages.

12. **What to diagram**
   - A good architecture diagram would show:
     - two separate top-level clouds
     - within AWS: IAM as the root security layer feeding Lake Formation, Glue, and EMR
     - within Azure: Storage Account and Service Principal feeding Databricks, with MySQL as a supporting service
     - CI/testing as an orthogonal lane validating the module compositions
   - The important idea is that this repo provisions **foundational infrastructure modules** for a data lake, not the data pipelines or applications that run on top of them.

Main entry points: [`aws_datalake`](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake) (AWS Root), [`azure_datalake`](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake) (Azure Root), [`aws_datalake/modules/iam`](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/iam) (IAM), [`.kitchen.yml`](https://github.com/segmentio/terraform-segment-data-lakes/blob/master/.kitchen.yml) (Test Kitchen)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_aws["AWS Data Lake"]
  node_aws_root["AWS Root<br/>terraform root"]
  node_aws_iam["IAM<br/>terraform module"]
  node_aws_lakeformation["Lake Formation<br/>terraform module"]
  node_aws_glue["Glue<br/>terraform module"]
  node_aws_emr["EMR<br/>terraform module"]
end

subgraph group_azure["Azure Data Lake"]
  node_azure_root["Azure Root<br/>terraform root"]
  node_azure_storage[("Storage<br/>terraform module")]
  node_azure_sp(("Service Principal<br/>terraform module"))
  node_azure_databricks["Databricks<br/>terraform module"]
  node_azure_mysql[("MySQL<br/>terraform module")]
end

subgraph group_delivery["Validation"]
  node_aws_fixture["AWS Fixture<br/>test fixture<br/>[aws_main.tf]"]
  node_azure_fixture["Azure Fixture<br/>test fixture<br/>[azure_main.tf]"]
  node_kitchen["Test Kitchen<br/>ruby test harness<br/>[.kitchen.yml]"]
  node_buildkite["Buildkite CI<br/>ci pipeline<br/>[pipeline.yml]"]
end

node_repo["Terraform Repo<br/>infrastructure repo"]

node_repo -->|"contains"| node_aws_root
node_repo -->|"contains"| node_azure_root
node_aws_root -->|"composes"| node_aws_iam
node_aws_root -->|"composes"| node_aws_lakeformation
node_aws_root -->|"composes"| node_aws_glue
node_aws_root -->|"composes"| node_aws_emr
node_aws_iam -->|"feeds"| node_aws_lakeformation
node_aws_iam -->|"authorizes"| node_aws_glue
node_aws_iam -->|"authorizes"| node_aws_emr
node_aws_fixture -.->|"wires"| node_aws_root
node_azure_root -->|"composes"| node_azure_storage
node_azure_root -->|"composes"| node_azure_sp
node_azure_root -->|"composes"| node_azure_databricks
node_azure_root -->|"composes"| node_azure_mysql
node_azure_storage -->|"supports"| node_azure_databricks
node_azure_sp -->|"authenticates"| node_azure_databricks
node_azure_fixture -.->|"wires"| node_azure_root
node_kitchen -.->|"runs"| node_aws_fixture
node_kitchen -.->|"runs"| node_azure_fixture
node_buildkite -.->|"invokes"| node_kitchen
node_buildkite -.->|"validates"| node_repo

click node_aws_root "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake"
click node_aws_iam "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/iam"
click node_aws_lakeformation "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/lakeformation"
click node_aws_glue "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/glue"
click node_aws_emr "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/emr"
click node_aws_fixture "https://github.com/segmentio/terraform-segment-data-lakes/blob/master/aws_datalake/test/test_fixture/aws_main.tf"
click node_azure_root "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake"
click node_azure_storage "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/StorageAccount"
click node_azure_sp "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/serviceprincipal"
click node_azure_databricks "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/databricks"
click node_azure_mysql "https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/mysql"
click node_azure_fixture "https://github.com/segmentio/terraform-segment-data-lakes/blob/master/azure_datalake/test/test_fixture/azure_main.tf"
click node_kitchen "https://github.com/segmentio/terraform-segment-data-lakes/blob/master/.kitchen.yml"
click node_buildkite "https://github.com/segmentio/terraform-segment-data-lakes/blob/master/.buildkite/pipeline.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_aws_root,node_aws_iam,node_aws_lakeformation,node_aws_glue,node_aws_emr toneBlue
class node_azure_root,node_azure_storage,node_azure_sp,node_azure_databricks,node_azure_mysql toneAmber
class node_aws_fixture,node_azure_fixture,node_kitchen,node_buildkite toneMint
class node_repo toneNeutral
```

## Components

### AWS Data Lake

Terraform modules for the AWS Segment data lake boundary.

- [AWS Root](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake): terraform root · AWS-specific module collection and test fixture. · `aws_datalake`
- [IAM](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/iam): terraform module · Identity, permissions, and cross-account access foundation. · `aws_datalake/modules/iam`
- [Lake Formation](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/lakeformation): terraform module · Governance and access control for lake assets. · `aws_datalake/modules/lakeformation`
- [Glue](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/glue): terraform module · Catalog and metadata services for the AWS lake. · `aws_datalake/modules/glue`
- [EMR](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/aws_datalake/modules/emr): terraform module · Managed processing layer for distributed lake workloads. · `aws_datalake/modules/emr`

### Azure Data Lake

Terraform modules for the Azure Segment data lake boundary.

- [Azure Root](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake): terraform root · Azure-specific module collection and test fixture. · `azure_datalake`
- [Storage](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/StorageAccount): terraform module · Durable lake storage boundary on Azure. · `azure_datalake/modules/StorageAccount`
- [Service Principal](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/serviceprincipal): terraform module · Identity and automation access for Azure resources. · `azure_datalake/modules/serviceprincipal`
- [Databricks](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/databricks): terraform module · Analytics workspace and compute integration. · `azure_datalake/modules/databricks`
- [MySQL](https://github.com/segmentio/terraform-segment-data-lakes/tree/master/azure_datalake/modules/mysql): terraform module · Supporting relational service adjacent to the lake platform. · `azure_datalake/modules/mysql`

### Validation

Integration testing and CI delivery for module composition.

- [AWS Fixture](https://github.com/segmentio/terraform-segment-data-lakes/blob/master/aws_datalake/test/test_fixture/aws_main.tf): test fixture · Minimal composed AWS environment used for integration validation. · `aws_datalake/test/test_fixture/aws_main.tf`
- [Azure Fixture](https://github.com/segmentio/terraform-segment-data-lakes/blob/master/azure_datalake/test/test_fixture/azure_main.tf): test fixture · Minimal composed Azure environment used for integration validation. · `azure_datalake/test/test_fixture/azure_main.tf`
- [Test Kitchen](https://github.com/segmentio/terraform-segment-data-lakes/blob/master/.kitchen.yml): ruby test harness · Ruby-based integration test orchestration for Terraform module fixtures. · `.kitchen.yml`
- [Buildkite CI](https://github.com/segmentio/terraform-segment-data-lakes/blob/master/.buildkite/pipeline.yml): ci pipeline · Pipeline used to validate module changes and fixture compositions. · `.buildkite/pipeline.yml`

### Other components

- **Terraform Repo**: infrastructure repo · Reusable Terraform modules for provisioning Segment data lake foundations across AWS and Azure.

## Connections

- Terraform Repo → AWS Root: contains · Repository includes the AWS Terraform root.
- Terraform Repo → Azure Root: contains · Repository includes the Azure Terraform root.
- AWS Root → IAM: composes · AWS root is built from IAM foundation module.
- AWS Root → Lake Formation: composes · AWS root is built from governance module.
- AWS Root → Glue: composes · AWS root is built from catalog module.
- AWS Root → EMR: composes · AWS root is built from processing module.
- IAM → Lake Formation: feeds · IAM provides the security basis for governance permissions.
- IAM → Glue: authorizes · IAM supports catalog service access.
- IAM → EMR: authorizes · IAM supports EMR compute access.
- AWS Fixture → AWS Root: wires · Test fixture assembles the AWS module composition.
- Azure Root → Storage: composes · Azure root is built around durable storage.
- Azure Root → Service Principal: composes · Azure root includes service principal identity.
- Azure Root → Databricks: composes · Azure root includes Databricks workspace integration.
- Azure Root → MySQL: composes · Azure root includes the supporting relational service.
- Storage → Databricks: supports · Storage is the data boundary consumed by Databricks.
- Service Principal → Databricks: authenticates · Service principal enables workspace and storage access.
- Azure Fixture → Azure Root: wires · Test fixture assembles the Azure module composition.
- Test Kitchen → AWS Fixture: runs · Test Kitchen executes AWS integration fixtures.
- Test Kitchen → Azure Fixture: runs · Test Kitchen executes Azure integration fixtures.
- Buildkite CI → Test Kitchen: invokes · CI pipeline runs the Ruby-based integration test harness.
- Buildkite CI → Terraform Repo: validates · Buildkite validates repository changes before downstream use.