# spirom/learningspark architecture

> Architecture diagram of the GitHub repository spirom/learningspark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spirom/learningspark
- Repository: https://github.com/spirom/learningspark

## Overview

LearningSpark is a collection of independent, runnable Scala and Java examples—not a single application pipeline. A learner reads an example, runs its main method, and inspects the Spark output. The examples cover RDD operations, typed and untyped data processing, SQL/Hive, streaming, and GraphX. The graph groups related demonstrations rather than implying they call one another. Sampled sources show specific helper and streaming relationships; other capability links are based on the README’s package descriptions.

Main entry points: [`src/main/scala/dataframe/DatasetConversion.scala`](https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/DatasetConversion.scala) (DataFrame Conversion), [`src/main/scala/streaming/CustomReceiver.scala`](https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/CustomReceiver.scala) (Custom Receiver), [`src/main/scala/Ex1_SimpleRDD.scala`](https://github.com/spirom/learningspark/blob/master/src/main/scala/Ex1_SimpleRDD.scala) (RDD Examples), [`src/main/scala/special`](https://github.com/spirom/learningspark/tree/master/src/main/scala/special) (Special RDDs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_rdd["RDD Processing"]
  node_rdd_examples["RDD Examples"]
  node_special_rdd["Special RDDs"]
  node_pair_rdd["Pair RDD<br/>[PairRDD.scala]"]
end

subgraph group_structured["Structured Data"]
  node_df_conversion["DataFrame Conversion"]
  node_df_schema["Rows And Schema"]
  node_df_select["DataFrame Selection<br/>[Select.scala]"]
  node_dataset_complex["Complex Datasets<br/>[ComplexType.scala]"]
  node_dataset_partition["Dataset Partitioning<br/>[PartitionBy.scala]"]
  node_partition_util["Table Hierarchy Utility"]
end

subgraph group_sql["SQL And Hive"]
  node_sql_examples["SQL Examples"]
  node_hive_partition["Hive Partitioned Tables"]
  node_hive_udaf["Hive Sales UDAF"]
end

subgraph group_streaming["Streaming"]
  node_custom_receiver["Custom Receiver"]
  node_full_monitor["Full Monitoring"]
  node_structured_stream["Structured Aggregation"]
  node_csv_generator["CSV Stream Generator"]
end

subgraph group_graph["Graph Processing"]
  node_graphx["Second-Degree Neighbors"]
end

node_learner(("Learner"))
node_spark{{"Apache Spark"}}

node_learner -->|"runs example"| node_rdd_examples
node_learner -->|"runs examples"| node_special_rdd
node_learner -->|"runs example"| node_df_conversion
node_learner -->|"runs examples"| node_sql_examples
node_learner -->|"runs example"| node_custom_receiver
node_learner -->|"runs example"| node_graphx
node_rdd_examples -->|"demonstrates"| node_spark
node_special_rdd -->|"demonstrates"| node_spark
node_df_conversion -->|"demonstrates"| node_spark
node_sql_examples -->|"demonstrates"| node_spark
node_custom_receiver -->|"demonstrates"| node_spark
node_graphx -->|"demonstrates"| node_spark
node_df_conversion -->|"uses selection"| node_df_select
node_dataset_partition -->|"calls utility"| node_partition_util
node_hive_partition -->|"calls utility"| node_partition_util
node_full_monitor -->|"registers receiver"| node_custom_receiver
node_structured_stream -->|"creates files"| node_csv_generator

click node_rdd_examples "https://github.com/spirom/learningspark/blob/master/src/main/scala/Ex1_SimpleRDD.scala"
click node_special_rdd "https://github.com/spirom/learningspark/tree/master/src/main/scala/special"
click node_pair_rdd "https://github.com/spirom/learningspark/blob/master/src/main/scala/special/PairRDD.scala"
click node_df_conversion "https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/DatasetConversion.scala"
click node_df_schema "https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/FromRowsAndSchema.scala"
click node_df_select "https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/Select.scala"
click node_dataset_complex "https://github.com/spirom/learningspark/blob/master/src/main/scala/dataset/ComplexType.scala"
click node_dataset_partition "https://github.com/spirom/learningspark/blob/master/src/main/scala/dataset/PartitionBy.scala"
click node_partition_util "https://github.com/spirom/learningspark/blob/master/src/main/scala/utils/PartitionedTableHierarchy.scala"
click node_sql_examples "https://github.com/spirom/learningspark/tree/master/src/main/scala/sql"
click node_hive_partition "https://github.com/spirom/learningspark/blob/master/src/main/scala/hiveql/PartitionedTable.scala"
click node_hive_udaf "https://github.com/spirom/learningspark/blob/master/src/main/java/hiveql/SumLargeSalesUDAF.java"
click node_custom_receiver "https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/CustomReceiver.scala"
click node_full_monitor "https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/FullMonitoring.scala"
click node_structured_stream "https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/structured/BasicAggregation.scala"
click node_csv_generator "https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/util/CSVFileStreamGenerator.scala"
click node_graphx "https://github.com/spirom/learningspark/blob/master/src/main/scala/graphx/SecondDegreeNeighbors.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rdd_examples,node_special_rdd,node_pair_rdd toneBlue
class node_df_conversion,node_df_schema,node_df_select,node_dataset_complex,node_dataset_partition,node_partition_util toneAmber
class node_sql_examples,node_hive_partition,node_hive_udaf toneMint
class node_custom_receiver,node_full_monitor,node_structured_stream,node_csv_generator toneRose
class node_graphx,node_learner,node_spark toneIndigo
```

## Components

### RDD Processing

- [RDD Examples](https://github.com/spirom/learningspark/blob/master/src/main/scala/Ex1_SimpleRDD.scala): `src/main/scala/Ex1_SimpleRDD.scala`
- [Special RDDs](https://github.com/spirom/learningspark/tree/master/src/main/scala/special): `src/main/scala/special`
- [Pair RDD](https://github.com/spirom/learningspark/blob/master/src/main/scala/special/PairRDD.scala): `src/main/scala/special/PairRDD.scala`

### Structured Data

- [DataFrame Conversion](https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/DatasetConversion.scala): `src/main/scala/dataframe/DatasetConversion.scala`
- [Rows And Schema](https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/FromRowsAndSchema.scala): `src/main/scala/dataframe/FromRowsAndSchema.scala`
- [DataFrame Selection](https://github.com/spirom/learningspark/blob/master/src/main/scala/dataframe/Select.scala): `src/main/scala/dataframe/Select.scala`
- [Complex Datasets](https://github.com/spirom/learningspark/blob/master/src/main/scala/dataset/ComplexType.scala): `src/main/scala/dataset/ComplexType.scala`
- [Dataset Partitioning](https://github.com/spirom/learningspark/blob/master/src/main/scala/dataset/PartitionBy.scala): `src/main/scala/dataset/PartitionBy.scala`
- [Table Hierarchy Utility](https://github.com/spirom/learningspark/blob/master/src/main/scala/utils/PartitionedTableHierarchy.scala): `src/main/scala/utils/PartitionedTableHierarchy.scala`

### SQL And Hive

- [SQL Examples](https://github.com/spirom/learningspark/tree/master/src/main/scala/sql): `src/main/scala/sql`
- [Hive Partitioned Tables](https://github.com/spirom/learningspark/blob/master/src/main/scala/hiveql/PartitionedTable.scala): `src/main/scala/hiveql/PartitionedTable.scala`
- [Hive Sales UDAF](https://github.com/spirom/learningspark/blob/master/src/main/java/hiveql/SumLargeSalesUDAF.java): `src/main/java/hiveql/SumLargeSalesUDAF.java`

### Streaming

- [Custom Receiver](https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/CustomReceiver.scala): `src/main/scala/streaming/CustomReceiver.scala`
- [Full Monitoring](https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/FullMonitoring.scala): `src/main/scala/streaming/FullMonitoring.scala`
- [Structured Aggregation](https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/structured/BasicAggregation.scala): `src/main/scala/streaming/structured/BasicAggregation.scala`
- [CSV Stream Generator](https://github.com/spirom/learningspark/blob/master/src/main/scala/streaming/util/CSVFileStreamGenerator.scala): `src/main/scala/streaming/util/CSVFileStreamGenerator.scala`

### Graph Processing

- [Second-Degree Neighbors](https://github.com/spirom/learningspark/blob/master/src/main/scala/graphx/SecondDegreeNeighbors.scala): `src/main/scala/graphx/SecondDegreeNeighbors.scala`

### Other components

- **Learner**
- **Apache Spark**

## Connections

- Learner → RDD Examples: runs example
- Learner → Special RDDs: runs examples
- Learner → DataFrame Conversion: runs example
- Learner → SQL Examples: runs examples
- Learner → Custom Receiver: runs example
- Learner → Second-Degree Neighbors: runs example
- RDD Examples → Apache Spark: demonstrates
- Special RDDs → Apache Spark: demonstrates
- DataFrame Conversion → Apache Spark: demonstrates
- SQL Examples → Apache Spark: demonstrates
- Custom Receiver → Apache Spark: demonstrates
- Second-Degree Neighbors → Apache Spark: demonstrates
- DataFrame Conversion → DataFrame Selection: uses selection (evidence: `src/main/scala/dataframe/DatasetConversion.scala`)
- Dataset Partitioning → Table Hierarchy Utility: calls utility (evidence: `src/main/scala/dataset/PartitionBy.scala`)
- Hive Partitioned Tables → Table Hierarchy Utility: calls utility (evidence: `src/main/scala/hiveql/PartitionedTable.scala`)
- Full Monitoring → Custom Receiver: registers receiver (evidence: `src/main/scala/streaming/FullMonitoring.scala`)
- Structured Aggregation → CSV Stream Generator: creates files (evidence: `src/main/scala/streaming/structured/BasicAggregation.scala`)
