How shivaramkothapally/data-engineer-handbook works
Major “Domains” or Components
• Educational Content (General Resources) – This includes files like README.md, books.md, communities.md, interviews.md, newsletters.md, and projects.md. They form the overarching guide and reference material for the data engineering journey.
• Bootcamp Modules – This is the heart of the repository. Under the “bootcamp” directory you have:
▪ An introductory module (introduction.md, software.md) which sets the stage.
▪ Multiple learning modules:
– “1-dimensional-data-modeling”
– “2-fact-data-modeling”
– “3-spark-fundamentals”
Each of these modules bucket diverse materials such as lecture labs, homework assignments, and resource slides.
• Module Subcomponents – Within each module, observe further subdivisions:
▪ Lecture-and-Lab materials (SQL scripts, notebooks, analytical queries, etc.)
▪ Homework assignments (with documentation and placeholders for deliverables)
▪ Supporting configuration and orchestration tools (Makefiles, docker-compose files, shell scripts, environment samples) that help set up environments.
▪ Source code areas (e.g., the “src” directory found in “3-spark-fundamentals”, with jobs, tests, etc.) which illustrate practical data jobs and pipelines.
Relationships and Interactions
▪ “Lecture-Lab” (depicted as a node containing SQL queries, notebooks)
▪ “Homework” (node showing the assignment and evaluation component)
▪ “Scripts/Configurations” (grouping Docker, Makefile, and initialization scripts)
▪ The introductory material feeds into and primes the learner for the specific modules.
▪ Configuration tools or orchestration (like Docker Compose) are prerequisites to running the lab and homework environments.
▪ Source code and tests (within “3-spark-fundamentals”) illustrate the iterative development process and validate learning outcomes.
• Outside the Bootcamp area, add nodes for “General Data Engineering Resources” which are referenced (books, communities, podcasts, etc.). These nodes can be marked as external inspirations/advice that feed into the overall learning approach.
Architectural Patterns and Technologies
• Although not an application with back-end servers or front-end interfaces, the repository shows distinct modular patterns:
▪ Separation of concerns: Each bootcamp module is self-contained, indicating disciplined content segmentation.
▪ Environment orchestration: Presence of docker-compose files and scripts reflects the use of containerization for environment reproducibility.
▪ Testing and CI (via src/tests) in “3-spark-fundamentals” demonstrates an emphasis on validated learning components.
• Technologies to note include SQL (for data modeling and labs), Docker (for containerized environments), Python (in the Spark fundamentals module), and Jupyter Notebooks (for interactive lab sessions).
Putting It All Together
• Branch out to “Introductory Resources” linking to bootcamp software guidelines and getting-started documents.
▪ For 1-dimensional-data-modeling: Show nodes for configuration (docker-compose.yml, Makefile), lecture labs (SQL files), and homework.
▪ For 2-fact-data-modeling: Similarly, highlight SQL scripts, lectures, and homework components.
▪ For 3-spark-fundamentals: Illustrate nodes for Jupyter Notebooks, source (jobs and tests), and associated data files.