How richiebachala/ai-based-data-engineering works
This repository is companion code for a book, not a single deployed application: engineers explore chapter-sized examples around the shared OpsPulse case study. The documented quickstart generates the dataset, then runs a readiness checklist; other chapters demonstrate separate capabilities from ingestion and SQL generation to retrieval, orchestration, governance, and observability. The graph preserves these as distinct example subsystems rather than implying one integrated runtime. Relationships are limited to the README and sampled source excerpts; many chapter implementations were not sampled.
Main entry points: opspulse_generator.py Dataset generator · documentation_pipeline.py Documentation pipeline · 04_orchestrator_workers.py Workflow patterns · 02_routing.py Workflow routing
Components 34
OpsPulse data
- Dataset generator
setup/opspulse_generator.py - Local DuckDB
- Snowflake
- Readiness checklist
ch01_foundations/ai_readiness_checklist.py - OpenLineage
ch02_modern_data_platform/lineage_openlineage.py - MCP introduction
ch02_modern_data_platform/mcp_server_intro.py
Ingestion and modeling
- Schema profiling
ch08_ingestion/schema_profiling.py - Documentation pipeline
ch08_ingestion/documentation_pipeline.py - Document extraction
ch08_ingestion/document_extraction.sql - dbt test generation
ch09_sql_generation/dbt_test_generation.py
AI workflows
- Safe SQL generation
ch09_sql_generation/self_healing_pipeline.py - Structured prompting
ch04_structured_prompting/data_engineering_prompt.py - Prompt repair
ch04_structured_prompting/prompt_repair_loop.py - Structured outputs
ch04_structured_prompting/structured_outputs_openai.py - Workflow patterns
ch03_workflow_patterns/04_orchestrator_workers.py - Workflow routing
ch03_workflow_patterns/02_routing.py
Context and tools
- Context sources
ch05_context_engineering/context_sources.py - Hybrid retrieval
ch05_context_engineering/hybrid_retrieval.py - Context shaping
ch05_context_engineering/context_shaping.py - Context provenance
ch05_context_engineering/provenance.py - Knowledge graph
ch06_graph_rag/knowledge_graph_networkx.py - GraphRAG search
ch06_graph_rag/graphrag_search.py - Neo4j export
ch06_graph_rag/neo4j_export.py - Catalog MCP server
ch07_mcp_tools/catalog_server.py - Analytics MCP server
ch07_mcp_tools/analytics_server.py - Operations MCP server
ch07_mcp_tools/operations_server.py
Operations and governance
- Pipeline triage
ch10_orchestration/pipeline_triage_dag.py - Approval gate
ch10_orchestration/approval_gate.py - Evaluation harness
ch11_observability/eval_harness.py - Observability tracing
ch11_observability/observability_tracing.py - Content scanning
ch12_governance/content_scanning.py - Masking policies
ch12_governance/masking_policies.sql - Use-case scoring
ch13_team_topology/use_case_scoring.py
Other components
- Engineer
Connections 8
- Engineer → Dataset generator: generates dataset
README.md - Dataset generator → Local DuckDB: loads locally
README.md - Dataset generator → Snowflake: loads remotely
README.md - Engineer → Readiness checklist: runs example
README.md - Documentation pipeline → Schema profiling: imports profiler
ch08_ingestion/documentation_pipeline.py - Engineer → Safe SQL generation: runs example
README.md - Workflow patterns → Engineer: returns impact report
ch03_workflow_patterns/04_orchestrator_workers.py - Workflow routing → Engineer: returns triage result
ch03_workflow_patterns/02_routing.py