How apress/data-engineering-with-generative-and-agentic-ai-on-aws works
This repository is a book companion containing deployable AWS data-engineering and generative-AI examples, rather than one integrated application. A prominent documented workflow is a user sending a data-engineering request to the AgentCore orchestrator, which routes work to specialist agents. Other distinct examples cover security and FinOps, lake operations, knowledge-graph RAG, and a reference data pipeline. The graph reflects documented routing and sampled code relationships; many projects are represented by their source directories because their runtime details were not sampled.
Main entry points: engine.py Iceberg monitor · ingest.py Vector ingestion · data_engineering_orchestrator_agent.py Request orchestrator · ingestion_stack.py Graph ingestion workflow
Components 26
Agent workflows
- Request orchestrator
12_agentic/agent/data-engineering-orchestrator-agent/agent/data_engineering_orchestrator_agent.py - Specialist agents
05_bigdata
Security and FinOps
- Bedrock audit trail
02_security/bedrock-audit-trail/cdk/stacks/audit_trail_stack.py - Agent cost tracing
02_security/finops/agent-run-cost-tracer/cdk/stacks/tracer_stack.py - Token budget breaker
02_security/finops/token-budget-circuit-breaker/cdk/stacks/budget_stack.py
Lake operations
- Data catalog
03_datalake/data-catalog/cdk/stacks/data_catalog_stack.py - Lake Formation bootstrap
03_datalake/lakeformation-bootstrap/cdk/stacks/lakeformation_bootstrap_stack.py - Iceberg monitor
03_datalake/iceberg-maintenance-agent/src/monitor/engine.py - Table analyzer
03_datalake/iceberg-maintenance-agent/src/monitor/analyzer.py - Monitor configuration
03_datalake/iceberg-maintenance-agent/src/monitor/config.py - Maintenance recommendations
03_datalake/iceberg-maintenance-agent/src/monitor/recommendations.py - Maintenance executor
03_datalake/iceberg-maintenance-agent/src/monitor/executor.py - Table metrics
03_datalake/iceberg-maintenance-agent/src/monitor/metrics.py
RAG systems
- Graph ingestion workflow
08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py - Neptune graph store
08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/neptune_stack.py - Knowledge graph agent
08_rag/knowledge-graph-agent/src/knowledge_graph_agent/agent/knowledge_graph_agent.py - Graph retrieval
08_rag/knowledge-graph-agent/src/knowledge_graph_agent/retrieval - Vector ingestion
08_rag/s3_vectors/src/s3vectors_rag/ingest.py - Vector configuration
08_rag/s3_vectors/src/s3vectors_rag/config.py - Vector logging
08_rag/s3_vectors/src/s3vectors_rag/logging_setup.py - S3 vector store
08_rag/s3_vectors/src/s3vectors_rag/store.py
Data pipelines
- Reference pipeline infrastructure
devops-data-engineering/reference-pipeline/cdk/stacks/glue-pipeline-stack.ts - Order ingestion job
devops-data-engineering/reference-pipeline/glue_jobs/ingest_orders.py - CDC ingestion
03_datalake/progressive-medallion-starter-kit/lambda/cdc_ingest.py - Structured logging helper
devops-data-engineering/labs/08-observability-and-rollback/logging.py
Other components
- Data engineer
Connections 14
- Data engineer → Request orchestrator: submits request
README.md - Request orchestrator → Specialist agents: routes requests
README.md - Iceberg monitor → Table analyzer: analyzes tables
03_datalake/iceberg-maintenance-agent/src/monitor/engine.py - Iceberg monitor → Maintenance recommendations: generates recommendations
03_datalake/iceberg-maintenance-agent/src/monitor/engine.py - Iceberg monitor → Maintenance executor: executes maintenance
03_datalake/iceberg-maintenance-agent/src/monitor/engine.py - Iceberg monitor → Table metrics: reports metrics
03_datalake/iceberg-maintenance-agent/src/monitor/engine.py - Iceberg monitor → Monitor configuration: uses configuration
03_datalake/iceberg-maintenance-agent/src/monitor/engine.py - Graph ingestion workflow → Neptune graph store: writes graph
08_rag/knowledge-graph-agent/infrastructure/cdk/stacks/ingestion_stack.py - Knowledge graph agent → Graph retrieval: retrieves context
- Vector ingestion → Vector configuration: reads settings
08_rag/s3_vectors/src/s3vectors_rag/ingest.py - Vector ingestion → S3 vector store: writes vectors
08_rag/s3_vectors/src/s3vectors_rag/ingest.py - Vector ingestion → Vector logging: logs ingestion
08_rag/s3_vectors/src/s3vectors_rag/ingest.py - Reference pipeline infrastructure → Order ingestion job: configures job
devops-data-engineering/reference-pipeline/cdk/stacks/glue-pipeline-stack.ts - CDC ingestion → Structured logging helper: emits logs
03_datalake/progressive-medallion-starter-kit/lambda/cdc_ingest.py