# j-suyash/data-engineering-bible architecture

> Architecture diagram of the GitHub repository j-suyash/data-engineering-bible, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/j-suyash/data-engineering-bible
- Repository: https://github.com/j-suyash/data-engineering-bible

## Overview

This repository is an offline, linked data-engineering interview knowledgebase, not an executable data platform. Its main user workflow is to choose a topic, practise its recitation prompts, and follow prerequisite/successor links; Obsidian, terminal search, and drill mode are documented ways to navigate it. The graph preserves the README’s topic taxonomy and documented prerequisite edges. Source excerpts were unavailable, so relationships are based on the README’s graph and usage descriptions only.

Main entry points: [`nodes/sql-semantics.md`](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-semantics.md) (SQL semantics), [`nodes/sql-ddl-dml-transactions.md`](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-ddl-dml-transactions.md) (DDL and transactions), [`nodes/dimensional-modeling.md`](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/dimensional-modeling.md) (Dimensional modeling), [`nodes/warehouse-architecture.md`](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/warehouse-architecture.md) (Warehouse architecture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_foundations["SQL and modeling"]
  node_sql_semantics["SQL semantics<br/>[sql-semantics.md]"]
  node_sql_joins["SQL joins<br/>[sql-joins-sets.md]"]
  node_sql_aggregation["SQL aggregation<br/>[sql-aggregation.md]"]
  node_sql_ddl["DDL and transactions"]
  node_sql_ctes["CTEs and subqueries"]
  node_sql_windows["Window functions"]
  node_sql_patterns["Query patterns"]
  node_sql_performance["SQL performance"]
  node_relational["Relational modeling"]
  node_dimensional["Dimensional modeling"]
  node_oltp_olap["OLTP and OLAP<br/>[oltp-vs-olap.md]"]
  node_scd["SCD history<br/>[scd-history.md]"]
  node_warehouse["Warehouse architecture"]
end

subgraph group_pipelines["Pipelines and streaming"]
  node_etl["ETL and ELT<br/>[etl-vs-elt.md]"]
  node_pipeline["Pipeline design<br/>[pipeline-design.md]"]
  node_airflow["Airflow orchestration"]
  node_dbt["dbt transformations"]
  node_cdc["CDC ingestion<br/>[cdc-ingestion.md]"]
  node_batch_stream["Batch and streaming"]
  node_kafka["Kafka messaging<br/>[kafka-messaging.md]"]
  node_lambda_kappa["Lambda and Kappa<br/>[lambda-kappa.md]"]
end

subgraph group_processing["Processing and storage"]
  node_distributed["Distributed systems"]
  node_spark["Spark core<br/>[spark-core.md]"]
  node_spark_tuning["Spark tuning<br/>[spark-tuning.md]"]
  node_hdfs["Hadoop HDFS<br/>[hadoop-hdfs.md]"]
  node_mapreduce["MapReduce<br/>[mapreduce.md]"]
  node_hive["Hive and Impala<br/>[hive-impala.md]"]
  node_file_formats["File formats<br/>[file-formats.md]"]
  node_partitioning["Partitioning layout"]
  node_table_formats["Lakehouse formats"]
  node_nosql["NoSQL stores<br/>[nosql-stores.md]"]
end

subgraph group_platforms["Cloud and operations"]
  node_cloud["Cloud fundamentals"]
  node_aws["AWS data stack<br/>[aws-data-stack.md]"]
end

subgraph group_adjacent["Adjacent skills"]
end

node_learner(("Interview learner"))

node_learner -->|"starts with"| node_sql_ddl
node_learner -->|"studies"| node_dimensional
node_learner -->|"studies"| node_scd
node_learner -->|"studies"| node_cloud
node_learner -->|"navigates topics"| node_warehouse
node_sql_semantics -->|"prerequisite"| node_sql_joins
node_sql_semantics -->|"prerequisite"| node_sql_aggregation
node_sql_semantics -->|"prerequisite"| node_sql_ddl
node_sql_semantics -->|"prerequisite"| node_sql_ctes
node_sql_joins -->|"prerequisite"| node_sql_aggregation
node_sql_aggregation -->|"prerequisite"| node_sql_windows
node_sql_windows -->|"prerequisite"| node_sql_patterns
node_sql_ctes -->|"prerequisite"| node_sql_patterns
node_sql_joins -->|"prerequisite"| node_sql_performance
node_sql_ddl -->|"prerequisite"| node_sql_performance
node_sql_ddl -->|"prerequisite"| node_relational
node_relational -->|"prerequisite"| node_dimensional
node_relational -->|"prerequisite"| node_oltp_olap
node_dimensional -->|"prerequisite"| node_scd
node_dimensional -->|"prerequisite"| node_warehouse
node_oltp_olap -->|"prerequisite"| node_warehouse
node_warehouse -->|"prerequisite"| node_etl
node_etl -->|"prerequisite"| node_pipeline
node_etl -->|"prerequisite"| node_dbt
node_pipeline -->|"prerequisite"| node_airflow
node_pipeline -->|"prerequisite"| node_batch_stream
node_scd -->|"prerequisite"| node_cdc
node_cdc -->|"prerequisite"| node_batch_stream
node_batch_stream -->|"prerequisite"| node_kafka
node_kafka -->|"prerequisite"| node_lambda_kappa
node_distributed -->|"prerequisite"| node_spark
node_spark -->|"prerequisite"| node_spark_tuning
node_hdfs -->|"prerequisite"| node_mapreduce
node_hdfs -->|"prerequisite"| node_hive
node_mapreduce -->|"prerequisite"| node_spark
node_file_formats -->|"prerequisite"| node_partitioning
node_file_formats -->|"prerequisite"| node_table_formats
node_partitioning -->|"prerequisite"| node_spark_tuning
node_distributed -->|"prerequisite"| node_nosql
node_cloud -->|"prerequisite"| node_aws

click node_sql_semantics "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-semantics.md"
click node_sql_joins "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-joins-sets.md"
click node_sql_aggregation "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-aggregation.md"
click node_sql_ddl "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-ddl-dml-transactions.md"
click node_sql_ctes "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-ctes-subqueries.md"
click node_sql_windows "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-window-functions.md"
click node_sql_patterns "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-query-patterns.md"
click node_sql_performance "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-performance-indexing.md"
click node_relational "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/relational-modeling.md"
click node_dimensional "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/dimensional-modeling.md"
click node_oltp_olap "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/oltp-vs-olap.md"
click node_scd "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/scd-history.md"
click node_warehouse "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/warehouse-architecture.md"
click node_etl "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/etl-vs-elt.md"
click node_pipeline "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/pipeline-design.md"
click node_airflow "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/orchestration-airflow.md"
click node_dbt "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/dbt-transformations.md"
click node_cdc "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/cdc-ingestion.md"
click node_batch_stream "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/batch-vs-streaming.md"
click node_kafka "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/kafka-messaging.md"
click node_lambda_kappa "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/lambda-kappa.md"
click node_distributed "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/distributed-systems.md"
click node_spark "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/spark-core.md"
click node_spark_tuning "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/spark-tuning.md"
click node_hdfs "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/hadoop-hdfs.md"
click node_mapreduce "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/mapreduce.md"
click node_hive "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/hive-impala.md"
click node_file_formats "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/file-formats.md"
click node_partitioning "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/partitioning-layout.md"
click node_table_formats "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/table-formats-lakehouse.md"
click node_nosql "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/nosql-stores.md"
click node_cloud "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/cloud-fundamentals.md"
click node_aws "https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/aws-data-stack.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sql_semantics,node_sql_joins,node_sql_aggregation,node_sql_ddl,node_sql_ctes,node_sql_windows,node_sql_patterns,node_sql_performance,node_relational,node_dimensional,node_oltp_olap,node_scd,node_warehouse,node_learner toneBlue
class node_etl,node_pipeline,node_airflow,node_dbt,node_cdc,node_batch_stream,node_kafka,node_lambda_kappa toneAmber
class node_distributed,node_spark,node_spark_tuning,node_hdfs,node_mapreduce,node_hive,node_file_formats,node_partitioning,node_table_formats,node_nosql toneMint
class node_cloud,node_aws toneRose
```

## Components

### SQL and modeling

- [SQL semantics](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-semantics.md): `nodes/sql-semantics.md`
- [SQL joins](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-joins-sets.md): `nodes/sql-joins-sets.md`
- [SQL aggregation](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-aggregation.md): `nodes/sql-aggregation.md`
- [DDL and transactions](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-ddl-dml-transactions.md): `nodes/sql-ddl-dml-transactions.md`
- [CTEs and subqueries](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-ctes-subqueries.md): `nodes/sql-ctes-subqueries.md`
- [Window functions](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-window-functions.md): `nodes/sql-window-functions.md`
- [Query patterns](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-query-patterns.md): `nodes/sql-query-patterns.md`
- [SQL performance](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/sql-performance-indexing.md): `nodes/sql-performance-indexing.md`
- [Relational modeling](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/relational-modeling.md): `nodes/relational-modeling.md`
- [Dimensional modeling](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/dimensional-modeling.md): `nodes/dimensional-modeling.md`
- [OLTP and OLAP](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/oltp-vs-olap.md): `nodes/oltp-vs-olap.md`
- [SCD history](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/scd-history.md): `nodes/scd-history.md`
- [Warehouse architecture](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/warehouse-architecture.md): `nodes/warehouse-architecture.md`

### Pipelines and streaming

- [ETL and ELT](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/etl-vs-elt.md): `nodes/etl-vs-elt.md`
- [Pipeline design](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/pipeline-design.md): `nodes/pipeline-design.md`
- [Airflow orchestration](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/orchestration-airflow.md): `nodes/orchestration-airflow.md`
- [dbt transformations](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/dbt-transformations.md): `nodes/dbt-transformations.md`
- [CDC ingestion](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/cdc-ingestion.md): `nodes/cdc-ingestion.md`
- [Batch and streaming](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/batch-vs-streaming.md): `nodes/batch-vs-streaming.md`
- [Kafka messaging](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/kafka-messaging.md): `nodes/kafka-messaging.md`
- [Lambda and Kappa](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/lambda-kappa.md): `nodes/lambda-kappa.md`

### Processing and storage

- [Distributed systems](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/distributed-systems.md): `nodes/distributed-systems.md`
- [Spark core](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/spark-core.md): `nodes/spark-core.md`
- [Spark tuning](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/spark-tuning.md): `nodes/spark-tuning.md`
- [Hadoop HDFS](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/hadoop-hdfs.md): `nodes/hadoop-hdfs.md`
- [MapReduce](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/mapreduce.md): `nodes/mapreduce.md`
- [Hive and Impala](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/hive-impala.md): `nodes/hive-impala.md`
- [File formats](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/file-formats.md): `nodes/file-formats.md`
- [Partitioning layout](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/partitioning-layout.md): `nodes/partitioning-layout.md`
- [Lakehouse formats](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/table-formats-lakehouse.md): `nodes/table-formats-lakehouse.md`
- [NoSQL stores](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/nosql-stores.md): `nodes/nosql-stores.md`

### Cloud and operations

- [Cloud fundamentals](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/cloud-fundamentals.md): `nodes/cloud-fundamentals.md`
- [AWS data stack](https://github.com/j-suyash/data-engineering-bible/blob/main/nodes/aws-data-stack.md): `nodes/aws-data-stack.md`

### Other components

- **Interview learner**

## Connections

- Interview learner → DDL and transactions: starts with (evidence: `README.md`)
- Interview learner → Dimensional modeling: studies (evidence: `README.md`)
- Interview learner → SCD history: studies (evidence: `README.md`)
- Interview learner → Cloud fundamentals: studies (evidence: `README.md`)
- Interview learner → Warehouse architecture: navigates topics (evidence: `README.md`)
- SQL semantics → SQL joins: prerequisite (evidence: `README.md`)
- SQL semantics → SQL aggregation: prerequisite (evidence: `README.md`)
- SQL semantics → DDL and transactions: prerequisite (evidence: `README.md`)
- SQL semantics → CTEs and subqueries: prerequisite (evidence: `README.md`)
- SQL joins → SQL aggregation: prerequisite (evidence: `README.md`)
- SQL aggregation → Window functions: prerequisite (evidence: `README.md`)
- Window functions → Query patterns: prerequisite (evidence: `README.md`)
- CTEs and subqueries → Query patterns: prerequisite (evidence: `README.md`)
- SQL joins → SQL performance: prerequisite (evidence: `README.md`)
- DDL and transactions → SQL performance: prerequisite (evidence: `README.md`)
- DDL and transactions → Relational modeling: prerequisite (evidence: `README.md`)
- Relational modeling → Dimensional modeling: prerequisite (evidence: `README.md`)
- Relational modeling → OLTP and OLAP: prerequisite (evidence: `README.md`)
- Dimensional modeling → SCD history: prerequisite (evidence: `README.md`)
- Dimensional modeling → Warehouse architecture: prerequisite (evidence: `README.md`)
- OLTP and OLAP → Warehouse architecture: prerequisite (evidence: `README.md`)
- Warehouse architecture → ETL and ELT: prerequisite (evidence: `README.md`)
- ETL and ELT → Pipeline design: prerequisite (evidence: `README.md`)
- ETL and ELT → dbt transformations: prerequisite (evidence: `README.md`)
- Pipeline design → Airflow orchestration: prerequisite (evidence: `README.md`)
- Pipeline design → Batch and streaming: prerequisite (evidence: `README.md`)
- SCD history → CDC ingestion: prerequisite (evidence: `README.md`)
- CDC ingestion → Batch and streaming: prerequisite (evidence: `README.md`)
- Batch and streaming → Kafka messaging: prerequisite (evidence: `README.md`)
- Kafka messaging → Lambda and Kappa: prerequisite (evidence: `README.md`)
- Distributed systems → Spark core: prerequisite (evidence: `README.md`)
- Spark core → Spark tuning: prerequisite (evidence: `README.md`)
- Hadoop HDFS → MapReduce: prerequisite (evidence: `README.md`)
- Hadoop HDFS → Hive and Impala: prerequisite (evidence: `README.md`)
- MapReduce → Spark core: prerequisite (evidence: `README.md`)
- File formats → Partitioning layout: prerequisite (evidence: `README.md`)
- File formats → Lakehouse formats: prerequisite (evidence: `README.md`)
- Partitioning layout → Spark tuning: prerequisite (evidence: `README.md`)
- Distributed systems → NoSQL stores: prerequisite (evidence: `README.md`)
- Cloud fundamentals → AWS data stack: prerequisite (evidence: `README.md`)
