# n0va9/cc5-hadoopmarkets architecture

> Architecture diagram of the GitHub repository n0va9/cc5-hadoopmarkets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/n0va9/cc5-hadoopmarkets
- Repository: https://github.com/n0va9/cc5-hadoopmarkets

## Overview

This is a Hadoop MapReduce batch application for aggregating public-procurement amounts by principal CPV code, then classifying and sorting those totals by amount. The main workflow takes a CSV input through a consolidation job, writes an intermediate SequenceFile, and feeds it to a second job that emits categorized text results. The graph includes the custom writable and Job 2’s sort, grouping, and partitioning components. Job wiring is evidenced by MainDriver; Hadoop’s runtime dispatch and storage behavior are represented only where its configuration makes them explicit.

Main entry points: [`cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java) (Main driver), [`cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java`](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java) (Category mapper), [`cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java`](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java) (CPV aggregator), [`cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java`](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java) (CSV mapper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Job orchestration"]
  node_main_driver["Main driver<br/>[MainDriver.java]"]
end

subgraph group_aggregation["CPV aggregation"]
  node_job1_mapper["CSV mapper<br/>[Job1Mapper.java]"]
  node_job1_reducer["CPV aggregator<br/>[Job1Reducer.java]"]
end

subgraph group_classification["Classification and sorting"]
  node_job2_mapper["Category mapper<br/>[Job2Mapper.java]"]
  node_job2_partitioner["Category partitioner"]
  node_cpv_comparator["Category grouping"]
  node_job2_reducer["Results formatter<br/>[Job2Reducer.java]"]
end

subgraph group_data_types["MapReduce data types"]
  node_montant_count["Amount and count"]
  node_amount_key["Sortable amount key"]
end

node_operator(("Batch operator"))
node_input_csv["Procurement CSV"]
node_intermediate["Job 1 SequenceFile"]
node_output_text["Categorized text output"]

node_operator -->|"runs"| node_main_driver
node_operator -->|"provides"| node_input_csv
node_main_driver -->|"reads"| node_input_csv
node_main_driver -->|"configures"| node_job1_mapper
node_main_driver -->|"configures"| node_job1_reducer
node_job1_mapper -->|"emits records"| node_job1_reducer
node_job1_mapper -->|"creates values"| node_montant_count
node_job1_reducer -->|"aggregates values"| node_montant_count
node_job1_reducer -->|"writes SequenceFile"| node_intermediate
node_main_driver -->|"sets Job 2 input"| node_intermediate
node_main_driver -->|"configures"| node_job2_mapper
node_intermediate -->|"feeds records"| node_job2_mapper
node_job2_mapper -->|"builds keys"| node_amount_key
node_job2_mapper -->|"dispatches keys"| node_job2_partitioner
node_job2_mapper -->|"sorts and groups"| node_cpv_comparator
node_main_driver -->|"configures"| node_job2_partitioner
node_main_driver -->|"configures"| node_cpv_comparator
node_main_driver -->|"configures"| node_job2_reducer
node_job2_mapper -->|"emits records"| node_job2_reducer
node_job2_reducer -->|"writes results"| node_output_text

click node_main_driver "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java"
click node_job1_mapper "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java"
click node_job1_reducer "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java"
click node_montant_count "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/MontantCountWritable.java"
click node_job2_mapper "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java"
click node_amount_key "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/AmountWritableComparable.java"
click node_job2_partitioner "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Partitioner.java"
click node_cpv_comparator "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/CPVGroupingComparator.java"
click node_job2_reducer "https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Reducer.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_driver toneBlue
class node_job1_mapper,node_job1_reducer toneAmber
class node_job2_mapper,node_job2_partitioner,node_cpv_comparator,node_job2_reducer toneMint
class node_montant_count,node_amount_key toneRose
class node_operator,node_input_csv,node_intermediate,node_output_text toneIndigo
```

## Components

### Job orchestration

- [Main driver](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java): `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`

### CPV aggregation

- [CSV mapper](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java): `cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java`
- [CPV aggregator](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java): `cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java`

### Classification and sorting

- [Category mapper](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java): `cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java`
- [Category partitioner](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Partitioner.java): `cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Partitioner.java`
- [Category grouping](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/CPVGroupingComparator.java): `cc-hadoop-marches-publics/src/main/java/com/project/job2/CPVGroupingComparator.java`
- [Results formatter](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Reducer.java): `cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Reducer.java`

### MapReduce data types

- [Amount and count](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job1/MontantCountWritable.java): `cc-hadoop-marches-publics/src/main/java/com/project/job1/MontantCountWritable.java`
- [Sortable amount key](https://github.com/n0va9/cc5-hadoopmarkets/blob/master/cc-hadoop-marches-publics/src/main/java/com/project/job2/AmountWritableComparable.java): `cc-hadoop-marches-publics/src/main/java/com/project/job2/AmountWritableComparable.java`

### Other components

- **Batch operator**
- **Procurement CSV**
- **Job 1 SequenceFile**
- **Categorized text output**

## Connections

- Batch operator → Main driver: runs (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Batch operator → Procurement CSV: provides
- Main driver → Procurement CSV: reads (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Main driver → CSV mapper: configures (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Main driver → CPV aggregator: configures (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- CSV mapper → CPV aggregator: emits records (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- CSV mapper → Amount and count: creates values (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Mapper.java`)
- CPV aggregator → Amount and count: aggregates values (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/job1/Job1Reducer.java`)
- CPV aggregator → Job 1 SequenceFile: writes SequenceFile
- Main driver → Job 1 SequenceFile: sets Job 2 input (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Main driver → Category mapper: configures (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Job 1 SequenceFile → Category mapper: feeds records
- Category mapper → Sortable amount key: builds keys (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/job2/Job2Mapper.java`)
- Category mapper → Category partitioner: dispatches keys (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Category mapper → Category grouping: sorts and groups (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Main driver → Category partitioner: configures (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Main driver → Category grouping: configures (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Main driver → Results formatter: configures (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Category mapper → Results formatter: emits records (evidence: `cc-hadoop-marches-publics/src/main/java/com/project/MainDriver.java`)
- Results formatter → Categorized text output: writes results
