# richiebachala/ai-based-data-engineering architecture

> Architecture diagram of the GitHub repository richiebachala/ai-based-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/richiebachala/ai-based-data-engineering
- Repository: https://github.com/richiebachala/ai-based-data-engineering

## Overview

This repository is companion code for a book, not a single deployed application: engineers explore chapter-sized examples around the shared OpsPulse case study. The documented quickstart generates the dataset, then runs a readiness checklist; other chapters demonstrate separate capabilities from ingestion and SQL generation to retrieval, orchestration, governance, and observability. The graph preserves these as distinct example subsystems rather than implying one integrated runtime. Relationships are limited to the README and sampled source excerpts; many chapter implementations were not sampled.

Main entry points: [`setup/opspulse_generator.py`](https://github.com/richiebachala/ai-based-data-engineering/blob/main/setup/opspulse_generator.py) (Dataset generator), [`ch08_ingestion/documentation_pipeline.py`](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/documentation_pipeline.py) (Documentation pipeline), [`ch03_workflow_patterns/04_orchestrator_workers.py`](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch03_workflow_patterns/04_orchestrator_workers.py) (Workflow patterns), [`ch03_workflow_patterns/02_routing.py`](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch03_workflow_patterns/02_routing.py) (Workflow routing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_dataset["OpsPulse data"]
  node_generator["Dataset generator"]
  node_localdb[("Local DuckDB")]
  node_snowflake[("Snowflake")]
  node_readiness["Readiness checklist"]
  node_lineage["OpenLineage"]
  node_mcp_intro["MCP introduction"]
end

subgraph group_ingestion["Ingestion and modeling"]
  node_profiling["Schema profiling"]
  node_documentation["Documentation pipeline"]
  node_extraction["Document extraction"]
  node_dbt_tests["dbt test generation"]
end

subgraph group_ai["AI workflows"]
  node_sql_generation["Safe SQL generation"]
  node_structured_prompt["Structured prompting"]
  node_prompt_repair["Prompt repair"]
  node_structured_outputs["Structured outputs"]
  node_workflow_patterns["Workflow patterns"]
  node_routing["Workflow routing<br/>[02_routing.py]"]
end

subgraph group_context["Context and tools"]
  node_context_sources["Context sources<br/>[context_sources.py]"]
  node_hybrid_retrieval["Hybrid retrieval"]
  node_context_shaping["Context shaping<br/>[context_shaping.py]"]
  node_provenance["Context provenance<br/>[provenance.py]"]
  node_graph["Knowledge graph"]
  node_graphrag["GraphRAG search<br/>[graphrag_search.py]"]
  node_neo4j["Neo4j export<br/>[neo4j_export.py]"]
  node_catalog_server["Catalog MCP server<br/>[catalog_server.py]"]
  node_analytics_server["Analytics MCP server"]
  node_operations_server["Operations MCP server"]
end

subgraph group_operations["Operations and governance"]
  node_triage["Pipeline triage"]
  node_approval["Approval gate<br/>[approval_gate.py]"]
  node_evals["Evaluation harness<br/>[eval_harness.py]"]
  node_tracing["Observability tracing"]
  node_scanning["Content scanning"]
  node_masking["Masking policies"]
  node_scoring["Use-case scoring"]
end

node_engineer(("Engineer"))

node_engineer -->|"generates dataset"| node_generator
node_generator -->|"loads locally"| node_localdb
node_generator -->|"loads remotely"| node_snowflake
node_engineer -->|"runs example"| node_readiness
node_documentation -->|"imports profiler"| node_profiling
node_engineer -->|"runs example"| node_sql_generation
node_workflow_patterns -->|"returns impact report"| node_engineer
node_routing -->|"returns triage result"| node_engineer

click node_generator "https://github.com/richiebachala/ai-based-data-engineering/blob/main/setup/opspulse_generator.py"
click node_readiness "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch01_foundations/ai_readiness_checklist.py"
click node_lineage "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch02_modern_data_platform/lineage_openlineage.py"
click node_mcp_intro "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch02_modern_data_platform/mcp_server_intro.py"
click node_profiling "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/schema_profiling.py"
click node_documentation "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/documentation_pipeline.py"
click node_extraction "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/document_extraction.sql"
click node_dbt_tests "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch09_sql_generation/dbt_test_generation.py"
click node_sql_generation "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch09_sql_generation/self_healing_pipeline.py"
click node_structured_prompt "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch04_structured_prompting/data_engineering_prompt.py"
click node_prompt_repair "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch04_structured_prompting/prompt_repair_loop.py"
click node_structured_outputs "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch04_structured_prompting/structured_outputs_openai.py"
click node_workflow_patterns "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch03_workflow_patterns/04_orchestrator_workers.py"
click node_routing "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch03_workflow_patterns/02_routing.py"
click node_context_sources "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/context_sources.py"
click node_hybrid_retrieval "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/hybrid_retrieval.py"
click node_context_shaping "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/context_shaping.py"
click node_provenance "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/provenance.py"
click node_graph "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch06_graph_rag/knowledge_graph_networkx.py"
click node_graphrag "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch06_graph_rag/graphrag_search.py"
click node_neo4j "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch06_graph_rag/neo4j_export.py"
click node_catalog_server "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch07_mcp_tools/catalog_server.py"
click node_analytics_server "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch07_mcp_tools/analytics_server.py"
click node_operations_server "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch07_mcp_tools/operations_server.py"
click node_triage "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch10_orchestration/pipeline_triage_dag.py"
click node_approval "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch10_orchestration/approval_gate.py"
click node_evals "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch11_observability/eval_harness.py"
click node_tracing "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch11_observability/observability_tracing.py"
click node_scanning "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch12_governance/content_scanning.py"
click node_masking "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch12_governance/masking_policies.sql"
click node_scoring "https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch13_team_topology/use_case_scoring.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_localdb,node_snowflake,node_readiness,node_lineage,node_mcp_intro toneBlue
class node_profiling,node_documentation,node_extraction,node_dbt_tests toneAmber
class node_sql_generation,node_structured_prompt,node_prompt_repair,node_structured_outputs,node_workflow_patterns,node_routing toneMint
class node_context_sources,node_hybrid_retrieval,node_context_shaping,node_provenance,node_graph,node_graphrag,node_neo4j,node_catalog_server,node_analytics_server,node_operations_server toneRose
class node_triage,node_approval,node_evals,node_tracing,node_scanning,node_masking,node_scoring,node_engineer toneIndigo
```

## Components

### OpsPulse data

- [Dataset generator](https://github.com/richiebachala/ai-based-data-engineering/blob/main/setup/opspulse_generator.py): `setup/opspulse_generator.py`
- **Local DuckDB**
- **Snowflake**
- [Readiness checklist](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch01_foundations/ai_readiness_checklist.py): `ch01_foundations/ai_readiness_checklist.py`
- [OpenLineage](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch02_modern_data_platform/lineage_openlineage.py): `ch02_modern_data_platform/lineage_openlineage.py`
- [MCP introduction](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch02_modern_data_platform/mcp_server_intro.py): `ch02_modern_data_platform/mcp_server_intro.py`

### Ingestion and modeling

- [Schema profiling](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/schema_profiling.py): `ch08_ingestion/schema_profiling.py`
- [Documentation pipeline](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/documentation_pipeline.py): `ch08_ingestion/documentation_pipeline.py`
- [Document extraction](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch08_ingestion/document_extraction.sql): `ch08_ingestion/document_extraction.sql`
- [dbt test generation](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch09_sql_generation/dbt_test_generation.py): `ch09_sql_generation/dbt_test_generation.py`

### AI workflows

- [Safe SQL generation](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch09_sql_generation/self_healing_pipeline.py): `ch09_sql_generation/self_healing_pipeline.py`
- [Structured prompting](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch04_structured_prompting/data_engineering_prompt.py): `ch04_structured_prompting/data_engineering_prompt.py`
- [Prompt repair](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch04_structured_prompting/prompt_repair_loop.py): `ch04_structured_prompting/prompt_repair_loop.py`
- [Structured outputs](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch04_structured_prompting/structured_outputs_openai.py): `ch04_structured_prompting/structured_outputs_openai.py`
- [Workflow patterns](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch03_workflow_patterns/04_orchestrator_workers.py): `ch03_workflow_patterns/04_orchestrator_workers.py`
- [Workflow routing](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch03_workflow_patterns/02_routing.py): `ch03_workflow_patterns/02_routing.py`

### Context and tools

- [Context sources](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/context_sources.py): `ch05_context_engineering/context_sources.py`
- [Hybrid retrieval](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/hybrid_retrieval.py): `ch05_context_engineering/hybrid_retrieval.py`
- [Context shaping](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/context_shaping.py): `ch05_context_engineering/context_shaping.py`
- [Context provenance](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch05_context_engineering/provenance.py): `ch05_context_engineering/provenance.py`
- [Knowledge graph](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch06_graph_rag/knowledge_graph_networkx.py): `ch06_graph_rag/knowledge_graph_networkx.py`
- [GraphRAG search](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch06_graph_rag/graphrag_search.py): `ch06_graph_rag/graphrag_search.py`
- [Neo4j export](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch06_graph_rag/neo4j_export.py): `ch06_graph_rag/neo4j_export.py`
- [Catalog MCP server](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch07_mcp_tools/catalog_server.py): `ch07_mcp_tools/catalog_server.py`
- [Analytics MCP server](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch07_mcp_tools/analytics_server.py): `ch07_mcp_tools/analytics_server.py`
- [Operations MCP server](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch07_mcp_tools/operations_server.py): `ch07_mcp_tools/operations_server.py`

### Operations and governance

- [Pipeline triage](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch10_orchestration/pipeline_triage_dag.py): `ch10_orchestration/pipeline_triage_dag.py`
- [Approval gate](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch10_orchestration/approval_gate.py): `ch10_orchestration/approval_gate.py`
- [Evaluation harness](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch11_observability/eval_harness.py): `ch11_observability/eval_harness.py`
- [Observability tracing](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch11_observability/observability_tracing.py): `ch11_observability/observability_tracing.py`
- [Content scanning](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch12_governance/content_scanning.py): `ch12_governance/content_scanning.py`
- [Masking policies](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch12_governance/masking_policies.sql): `ch12_governance/masking_policies.sql`
- [Use-case scoring](https://github.com/richiebachala/ai-based-data-engineering/blob/main/ch13_team_topology/use_case_scoring.py): `ch13_team_topology/use_case_scoring.py`

### Other components

- **Engineer**

## Connections

- Engineer → Dataset generator: generates dataset (evidence: `README.md`)
- Dataset generator → Local DuckDB: loads locally (evidence: `README.md`)
- Dataset generator → Snowflake: loads remotely (evidence: `README.md`)
- Engineer → Readiness checklist: runs example (evidence: `README.md`)
- Documentation pipeline → Schema profiling: imports profiler (evidence: `ch08_ingestion/documentation_pipeline.py`)
- Engineer → Safe SQL generation: runs example (evidence: `README.md`)
- Workflow patterns → Engineer: returns impact report (evidence: `ch03_workflow_patterns/04_orchestrator_workers.py`)
- Workflow routing → Engineer: returns triage result (evidence: `ch03_workflow_patterns/02_routing.py`)
