How n0va9/cc5-hadoopmarkets works
This is a Hadoop MapReduce batch application for aggregating public-procurement amounts by principal CPV code, then classifying and sorting those totals by amount. The main workflow takes a CSV input through a consolidation job, writes an intermediate SequenceFile, and feeds it to a second job that emits categorized text results. The graph includes the custom writable and Job 2’s sort, grouping, and partitioning components. Job wiring is evidenced by MainDriver; Hadoop’s runtime dispatch and storage behavior are represented only where its configuration makes them explicit.
Main entry points: MainDriver.java Main driver · Job2Mapper.java Category mapper · Job1Reducer.java CPV aggregator · Job1Mapper.java CSV mapper
Components 13
Job orchestration
- Main driver
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java
CPV aggregation
- CSV mapper
cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java - CPV aggregator
cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java
Classification and sorting
- Category mapper
cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java - Category partitioner
cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Partitioner.java - Category grouping
cc-hadoop-marches-publics/src/main/java/com/project/job2/CPVGroupingComparator.java - Results formatter
cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Reducer.java
MapReduce data types
- Amount and count
cc-hadoop-marches-publics/src/main/java/com/project/job1/MontantCountWritable.java - Sortable amount key
cc-hadoop-marches-publics/src/main/java/com/project/job2/AmountWritableComparable.java
Other components
- Batch operator
- Procurement CSV
- Job 1 SequenceFile
- Categorized text output
Connections 20
- Batch operator → Main driver: runs
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Batch operator → Procurement CSV: provides
- Main driver → Procurement CSV: reads
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Main driver → CSV mapper: configures
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Main driver → CPV aggregator: configures
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - CSV mapper → CPV aggregator: emits records
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - CSV mapper → Amount and count: creates values
cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java - CPV aggregator → Amount and count: aggregates values
cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java - CPV aggregator → Job 1 SequenceFile: writes SequenceFile
- Main driver → Job 1 SequenceFile: sets Job 2 input
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Main driver → Category mapper: configures
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Job 1 SequenceFile → Category mapper: feeds records
- Category mapper → Sortable amount key: builds keys
cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java - Category mapper → Category partitioner: dispatches keys
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Category mapper → Category grouping: sorts and groups
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Main driver → Category partitioner: configures
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Main driver → Category grouping: configures
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Main driver → Results formatter: configures
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Category mapper → Results formatter: emits records
cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java - Results formatter → Categorized text output: writes results