How andrez00/hadoop-coursera works
Project type and purpose
• This is not a typical “product” codebase but a set of educational Hadoop and Spark exercises (MapReduce word count, joins, PySpark pipelines).
• The README (“# Hadoop”) and the folder names Week4_Lesson1/2 and Week5_Lesson1/2/3 reveal the purpose: teaching fundamentals of Hadoop MapReduce (mappers, reducers, intermediate joins) and Spark joins.
File structure
• Top-level: README.md, About the Course.txt, Week4_Lesson1, Week4_Lesson2, Week5_Lesson1/2/3.
• Week4 covers classic Hadoop MapReduce jobs:
• wordcount_mapper.py / wordcount_reducer.py
• join1_mapper.py / join1_reducer.py and join2_mapper.py / join2_reducer.py with synthetic data generators.
• Week5 shifts to PySpark IPython setups (install notes) and join examples.
• No “src” or “backend/frontend” separation, but clear separation by lesson folders.
README for additional insights
• README is minimal (“# Hadoop”), but folder names and text files contain instructions and outputs showing Hadoop command lines, expected outputs, and Spark code snippets.
Elements
– Client / Notebook
– HDFS NameNode & DataNodes
– YARN ResourceManager & NodeManagers
– MapReduce Tasks (Mapper, Reducer)
– Spark Driver & Executors
– Input / Output Data Stores
– From client → job submission → ResourceManager
– From DataNodes → mappers → shuffle → reducers → output
– From HDFS → Spark executors → HDFS
– Blue for storage components (NameNode, DataNodes)
– Green for compute (mappers, reducers, executors)
– Orange for the client/notebook interface
Detailed guidance
• On each DataNode, show mapper and reducer containers for Lesson1 & Lesson2 jobs.
• At the left, represent the user/client environment where wordcount_mapper.py, join scripts, and spark notebooks reside.