How segmentio/terraform-segment-data-lakes works
1. What this repo is
- This is an infrastructure repository, not an application codebase.
- It contains Terraform modules for provisioning a Segment data lake in two cloud boundaries: AWS and Azure.
- The repo is organized as reusable modules plus example/test fixtures, so the architecture is “module composition” rather than a single monolithic stack.
2. Top-level split: AWS vs Azure
- The repository is cleanly divided into two independent roots:
- aws_datalake/
- azure_datalake/
- Each root has its own module set, README, and test fixture, which suggests the two clouds are deployed and validated separately.
3. AWS subsystem
- AWS is broken into four Terraform modules:
- iam for identities, permissions, and cross-account access
- lakeformation for data lake governance and permissions
- glue for data catalog / ETL-oriented metadata services
- emr for managed compute / processing
- These are the core AWS building blocks for a governed lake: access control, catalog/governance, metadata/transform services, and distributed processing.
4. AWS data/control flow
- The likely dependency order is:
- IAM establishes roles/policies and external trust relationships first.
- Lake Formation then uses those permissions to control access to data assets.
- Glue provides cataloging and metadata services that Lake Formation and consumers rely on.
- EMR consumes the IAM and governance setup to run Spark/Hadoop-style processing against the lake.
- In architecture terms, IAM is the foundational security boundary; the other modules sit on top of it.
5. Azure subsystem
- Azure is also split into four modules:
- StorageAccount for the lake storage layer
- serviceprincipal for identity and access
- databricks for the analytics/compute workspace
- mysql for a supporting relational service
- This is a different shape from AWS: Azure centers on storage + identity + Databricks compute, with MySQL as an adjacent dependency.
6. Azure data/control flow
- The typical sequence is:
- StorageAccount creates the durable data lake storage boundary.
- serviceprincipal creates the identity used for automation and access.
- databricks integrates compute/workspace resources with storage and identity.
- mysql provisions a relational dependency that appears to support the wider platform rather than the lake itself.
- The key architectural boundary here is the separation between storage, identity, and analytics workspace.
7. Module design and boundaries
- The repo favors small, purpose-built Terraform modules over a single environment stack.
- Each module has its own main.tf, variables.tf, and output.tf, which indicates:
- inputs are explicit
- outputs are used to wire modules together
- stateful concerns are separated per component
- This makes the architecture diagram naturally component-based: boxes for modules, arrows for outputs/inputs.
8. Testing structure
- Both cloud roots have a test/ directory with a test_fixture/ and integration test files.
- The AWS side includes a Ruby test file under test/integration/default/, and the repo has Gemfile, Gemfile.lock, and .kitchen.yml, which strongly suggests Test Kitchen / Ruby-based Terraform integration testing.
- The fixtures act as minimal composed environments to validate that modules can be wired together and applied successfully.
9. CI and delivery
- .buildkite/pipeline.yml shows this repo is validated in Buildkite CI.
- The architecture of delivery is therefore:
- commit Terraform/module changes
- CI runs validation/integration tests
- modules are consumed by downstream environment stacks
- This repo does not appear to deploy a live runtime by itself; it produces reusable infrastructure primitives.
10. Technology and runtime boundaries
- The primary implementation language is Terraform (HCL).
- Supporting tooling is Ruby for test orchestration and Buildkite for CI.
- The repo targets Terraform 0.12+ and explicitly notes AWS provider version constraints to avoid breaking changes, which is important for dependency compatibility in the architecture.
11. External/cloud service boundaries
- AWS side materially depends on:
- IAM
- Lake Formation
- Glue
- EMR
- Azure side materially depends on:
- Storage Account
- Databricks
- Service Principal / Azure AD identity
- MySQL
- These are the real external system boundaries the diagram should show, not internal code packages.
12. What to diagram
- A good architecture diagram would show:
- two separate top-level clouds
- within AWS: IAM as the root security layer feeding Lake Formation, Glue, and EMR
- within Azure: Storage Account and Service Principal feeding Databricks, with MySQL as a supporting service
- CI/testing as an orthogonal lane validating the module compositions
- The important idea is that this repo provisions foundational infrastructure modules for a data lake, not the data pipelines or applications that run on top of them.
Main entry points: aws_datalake AWS Root · azure_datalake Azure Root · iam IAM · .kitchen.yml Test Kitchen
Components 15
AWS Data Lake
Terraform modules for the AWS Segment data lake boundary.
- AWS Root · AWS-specific module collection and test fixture.
aws_datalake - IAM · Identity, permissions, and cross-account access foundation.
aws_datalake/modules/iam - Lake Formation · Governance and access control for lake assets.
aws_datalake/modules/lakeformation - Glue · Catalog and metadata services for the AWS lake.
aws_datalake/modules/glue - EMR · Managed processing layer for distributed lake workloads.
aws_datalake/modules/emr
Azure Data Lake
Terraform modules for the Azure Segment data lake boundary.
- Azure Root · Azure-specific module collection and test fixture.
azure_datalake - Storage · Durable lake storage boundary on Azure.
azure_datalake/modules/StorageAccount - Service Principal · Identity and automation access for Azure resources.
azure_datalake/modules/serviceprincipal - Databricks · Analytics workspace and compute integration.
azure_datalake/modules/databricks - MySQL · Supporting relational service adjacent to the lake platform.
azure_datalake/modules/mysql
Validation
Integration testing and CI delivery for module composition.
- AWS Fixture · Minimal composed AWS environment used for integration validation.
aws_datalake/test/test_fixture/aws_main.tf - Azure Fixture · Minimal composed Azure environment used for integration validation.
azure_datalake/test/test_fixture/azure_main.tf - Test Kitchen · Ruby-based integration test orchestration for Terraform module fixtures.
.kitchen.yml - Buildkite CI · Pipeline used to validate module changes and fixture compositions.
.buildkite/pipeline.yml
Other components
- Terraform Repo · Reusable Terraform modules for provisioning Segment data lake foundations across AWS and Azure.
Connections 21
- Terraform Repo → AWS Root: contains · Repository includes the AWS Terraform root.
- Terraform Repo → Azure Root: contains · Repository includes the Azure Terraform root.
- AWS Root → IAM: composes · AWS root is built from IAM foundation module.
- AWS Root → Lake Formation: composes · AWS root is built from governance module.
- AWS Root → Glue: composes · AWS root is built from catalog module.
- AWS Root → EMR: composes · AWS root is built from processing module.
- IAM → Lake Formation: feeds · IAM provides the security basis for governance permissions.
- IAM → Glue: authorizes · IAM supports catalog service access.
- IAM → EMR: authorizes · IAM supports EMR compute access.
- AWS Fixture → AWS Root: wires · Test fixture assembles the AWS module composition.
- Azure Root → Storage: composes · Azure root is built around durable storage.
- Azure Root → Service Principal: composes · Azure root includes service principal identity.
- Azure Root → Databricks: composes · Azure root includes Databricks workspace integration.
- Azure Root → MySQL: composes · Azure root includes the supporting relational service.
- Storage → Databricks: supports · Storage is the data boundary consumed by Databricks.
- Service Principal → Databricks: authenticates · Service principal enables workspace and storage access.
- Azure Fixture → Azure Root: wires · Test fixture assembles the Azure module composition.
- Test Kitchen → AWS Fixture: runs · Test Kitchen executes AWS integration fixtures.
- Test Kitchen → Azure Fixture: runs · Test Kitchen executes Azure integration fixtures.
- Buildkite CI → Test Kitchen: invokes · CI pipeline runs the Ruby-based integration test harness.
- Buildkite CI → Terraform Repo: validates · Buildkite validates repository changes before downstream use.