# apress/data-engineering-with-generative-and-agentic-ai-on-aws architecture

> Architecture diagram of the GitHub repository apress/data-engineering-with-generative-and-agentic-ai-on-aws, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws
- Repository: https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws

## Overview

This repository is a book companion containing deployable AWS data-engineering and generative-AI examples, rather than one integrated application. A prominent documented workflow is a user sending a data-engineering request to the AgentCore orchestrator, which routes work to specialist agents. Other distinct examples cover security and FinOps, lake operations, knowledge-graph RAG, and a reference data pipeline. The graph reflects documented routing and sampled code relationships; many projects are represented by their source directories because their runtime details were not sampled.

Main entry points: [`03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/engine.py) (Iceberg monitor), [`08_rag/s3_vectors/src/s3vectors_rag/ingest.py`](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/ingest.py) (Vector ingestion), [`12_agentic/agent/data-engineering-orchestrator-agent/agent/data_engineering_orchestrator_agent.py`](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/12_agentic/agent/data-engineering-orchestrator-agent/agent/data_engineering_orchestrator_agent.py) (Request orchestrator), [`08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py`](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py) (Graph ingestion workflow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_agents["Agent workflows"]
  node_orchestrator["Request orchestrator"]
  node_specialists["Specialist agents"]
end

subgraph group_security["Security and FinOps"]
  node_audit["Bedrock audit trail"]
  node_costtrace["Agent cost tracing<br/>[tracer_stack.py]"]
  node_budget["Token budget breaker<br/>[budget_stack.py]"]
end

subgraph group_lake["Lake operations"]
  node_catalog["Data catalog"]
  node_lakeform["Lake Formation bootstrap"]
  node_monitorengine["Iceberg monitor<br/>[engine.py]"]
  node_analyzer["Table analyzer<br/>[analyzer.py]"]
  node_monitorconfig["Monitor configuration<br/>[config.py]"]
  node_recommendations["Maintenance recommendations<br/>[recommendations.py]"]
  node_executor["Maintenance executor<br/>[executor.py]"]
  node_metrics["Table metrics<br/>[metrics.py]"]
end

subgraph group_rag["RAG systems"]
  node_kgingestion["Graph ingestion workflow<br/>[ingestion_stack.py]"]
  node_neptune[("Neptune graph store<br/>[neptune_stack.py]")]
  node_kgagent["Knowledge graph agent"]
  node_kgretrieval["Graph retrieval"]
  node_vectoringest["Vector ingestion<br/>[ingest.py]"]
  node_vectorconfig["Vector configuration<br/>[config.py]"]
  node_vectorlogging["Vector logging<br/>[logging_setup.py]"]
  node_vectorstore[("S3 vector store<br/>[store.py]")]
end

subgraph group_pipeline["Data pipelines"]
  node_reference["Reference pipeline infrastructure"]
  node_gluejob["Order ingestion job<br/>[ingest_orders.py]"]
  node_cdc["CDC ingestion<br/>[cdc_ingest.py]"]
  node_structuredlog["Structured logging helper<br/>[logging.py]"]
end

node_user(("Data engineer"))

node_user -->|"submits request"| node_orchestrator
node_orchestrator -->|"routes requests"| node_specialists
node_monitorengine -->|"analyzes tables"| node_analyzer
node_monitorengine -->|"generates recommendations"| node_recommendations
node_monitorengine -->|"executes maintenance"| node_executor
node_monitorengine -.->|"reports metrics"| node_metrics
node_monitorengine -->|"uses configuration"| node_monitorconfig
node_kgingestion -->|"writes graph"| node_neptune
node_kgagent -.->|"retrieves context"| node_kgretrieval
node_vectoringest -->|"reads settings"| node_vectorconfig
node_vectoringest -->|"writes vectors"| node_vectorstore
node_vectoringest -->|"logs ingestion"| node_vectorlogging
node_reference -->|"configures job"| node_gluejob
node_cdc -.->|"emits logs"| node_structuredlog

click node_orchestrator "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/12_agentic/agent/data-engineering-orchestrator-agent/agent/data_engineering_orchestrator_agent.py"
click node_specialists "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/tree/main/05_bigdata"
click node_audit "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/02_security/bedrock-audit-trail/cdk/stacks/audit_trail_stack.py"
click node_costtrace "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/02_security/finops/agent-run-cost-tracer/cdk/stacks/tracer_stack.py"
click node_budget "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/02_security/finops/token-budget-circuit-breaker/cdk/stacks/budget_stack.py"
click node_catalog "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/data-catalog/cdk/stacks/data_catalog_stack.py"
click node_lakeform "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/lakeformation-bootstrap/cdk/stacks/lakeformation_bootstrap_stack.py"
click node_monitorengine "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/engine.py"
click node_analyzer "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/analyzer.py"
click node_monitorconfig "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/config.py"
click node_recommendations "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/recommendations.py"
click node_executor "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/executor.py"
click node_metrics "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/metrics.py"
click node_kgingestion "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py"
click node_neptune "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/neptune_stack.py"
click node_kgagent "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/src/knowledge_graph_agent/agent/knowledge_graph_agent.py"
click node_kgretrieval "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/tree/main/08_rag/knowledge-graph-agent/src/knowledge_graph_agent/retrieval"
click node_vectoringest "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/ingest.py"
click node_vectorconfig "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/config.py"
click node_vectorlogging "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/logging_setup.py"
click node_vectorstore "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/store.py"
click node_reference "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/devops-data-engineering/reference-pipeline/cdk/stacks/glue-pipeline-stack.ts"
click node_gluejob "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/devops-data-engineering/reference-pipeline/glue_jobs/ingest_orders.py"
click node_cdc "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/progressive-medallion-starter-kit/lambda/cdc_ingest.py"
click node_structuredlog "https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/devops-data-engineering/labs/08-observability-and-rollback/logging.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_specialists toneBlue
class node_audit,node_costtrace,node_budget toneAmber
class node_catalog,node_lakeform,node_monitorengine,node_analyzer,node_monitorconfig,node_recommendations,node_executor,node_metrics toneMint
class node_kgingestion,node_neptune,node_kgagent,node_kgretrieval,node_vectoringest,node_vectorconfig,node_vectorlogging,node_vectorstore toneRose
class node_reference,node_gluejob,node_cdc,node_structuredlog,node_user toneIndigo
```

## Components

### Agent workflows

- [Request orchestrator](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/12_agentic/agent/data-engineering-orchestrator-agent/agent/data_engineering_orchestrator_agent.py): `12_agentic/agent/data-engineering-orchestrator-agent/agent/data_engineering_orchestrator_agent.py`
- [Specialist agents](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/tree/main/05_bigdata): `05_bigdata`

### Security and FinOps

- [Bedrock audit trail](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/02_security/bedrock-audit-trail/cdk/stacks/audit_trail_stack.py): `02_security/bedrock-audit-trail/cdk/stacks/audit_trail_stack.py`
- [Agent cost tracing](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/02_security/finops/agent-run-cost-tracer/cdk/stacks/tracer_stack.py): `02_security/finops/agent-run-cost-tracer/cdk/stacks/tracer_stack.py`
- [Token budget breaker](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/02_security/finops/token-budget-circuit-breaker/cdk/stacks/budget_stack.py): `02_security/finops/token-budget-circuit-breaker/cdk/stacks/budget_stack.py`

### Lake operations

- [Data catalog](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/data-catalog/cdk/stacks/data_catalog_stack.py): `03_datalake/data-catalog/cdk/stacks/data_catalog_stack.py`
- [Lake Formation bootstrap](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/lakeformation-bootstrap/cdk/stacks/lakeformation_bootstrap_stack.py): `03_datalake/lakeformation-bootstrap/cdk/stacks/lakeformation_bootstrap_stack.py`
- [Iceberg monitor](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/engine.py): `03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`
- [Table analyzer](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/analyzer.py): `03_datalake/iceberg-maintenance-agent/src/monitor/analyzer.py`
- [Monitor configuration](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/config.py): `03_datalake/iceberg-maintenance-agent/src/monitor/config.py`
- [Maintenance recommendations](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/recommendations.py): `03_datalake/iceberg-maintenance-agent/src/monitor/recommendations.py`
- [Maintenance executor](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/executor.py): `03_datalake/iceberg-maintenance-agent/src/monitor/executor.py`
- [Table metrics](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/iceberg-maintenance-agent/src/monitor/metrics.py): `03_datalake/iceberg-maintenance-agent/src/monitor/metrics.py`

### RAG systems

- [Graph ingestion workflow](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py): `08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py`
- [Neptune graph store](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/neptune_stack.py): `08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/neptune_stack.py`
- [Knowledge graph agent](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/knowledge-graph-agent/src/knowledge_graph_agent/agent/knowledge_graph_agent.py): `08_rag/knowledge-graph-agent/src/knowledge_graph_agent/agent/knowledge_graph_agent.py`
- [Graph retrieval](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/tree/main/08_rag/knowledge-graph-agent/src/knowledge_graph_agent/retrieval): `08_rag/knowledge-graph-agent/src/knowledge_graph_agent/retrieval`
- [Vector ingestion](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/ingest.py): `08_rag/s3_vectors/src/s3vectors_rag/ingest.py`
- [Vector configuration](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/config.py): `08_rag/s3_vectors/src/s3vectors_rag/config.py`
- [Vector logging](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/logging_setup.py): `08_rag/s3_vectors/src/s3vectors_rag/logging_setup.py`
- [S3 vector store](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/08_rag/s3_vectors/src/s3vectors_rag/store.py): `08_rag/s3_vectors/src/s3vectors_rag/store.py`

### Data pipelines

- [Reference pipeline infrastructure](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/devops-data-engineering/reference-pipeline/cdk/stacks/glue-pipeline-stack.ts): `devops-data-engineering/reference-pipeline/cdk/stacks/glue-pipeline-stack.ts`
- [Order ingestion job](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/devops-data-engineering/reference-pipeline/glue_jobs/ingest_orders.py): `devops-data-engineering/reference-pipeline/glue_jobs/ingest_orders.py`
- [CDC ingestion](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/03_datalake/progressive-medallion-starter-kit/lambda/cdc_ingest.py): `03_datalake/progressive-medallion-starter-kit/lambda/cdc_ingest.py`
- [Structured logging helper](https://github.com/apress/data-engineering-with-generative-and-agentic-ai-on-aws/blob/main/devops-data-engineering/labs/08-observability-and-rollback/logging.py): `devops-data-engineering/labs/08-observability-and-rollback/logging.py`

### Other components

- **Data engineer**

## Connections

- Data engineer → Request orchestrator: submits request (evidence: `README.md`)
- Request orchestrator → Specialist agents: routes requests (evidence: `README.md`)
- Iceberg monitor → Table analyzer: analyzes tables (evidence: `03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`)
- Iceberg monitor → Maintenance recommendations: generates recommendations (evidence: `03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`)
- Iceberg monitor → Maintenance executor: executes maintenance (evidence: `03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`)
- Iceberg monitor → Table metrics: reports metrics (evidence: `03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`)
- Iceberg monitor → Monitor configuration: uses configuration (evidence: `03_datalake/iceberg-maintenance-agent/src/monitor/engine.py`)
- Graph ingestion workflow → Neptune graph store: writes graph (evidence: `08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py`)
- Knowledge graph agent → Graph retrieval: retrieves context
- Vector ingestion → Vector configuration: reads settings (evidence: `08_rag/s3_vectors/src/s3vectors_rag/ingest.py`)
- Vector ingestion → S3 vector store: writes vectors (evidence: `08_rag/s3_vectors/src/s3vectors_rag/ingest.py`)
- Vector ingestion → Vector logging: logs ingestion (evidence: `08_rag/s3_vectors/src/s3vectors_rag/ingest.py`)
- Reference pipeline infrastructure → Order ingestion job: configures job (evidence: `devops-data-engineering/reference-pipeline/cdk/stacks/glue-pipeline-stack.ts`)
- CDC ingestion → Structured logging helper: emits logs (evidence: `03_datalake/progressive-medallion-starter-kit/lambda/cdc_ingest.py`)
