How spirom/learningspark works
LearningSpark is a collection of independent, runnable Scala and Java examples—not a single application pipeline. A learner reads an example, runs its main method, and inspects the Spark output. The examples cover RDD operations, typed and untyped data processing, SQL/Hive, streaming, and GraphX. The graph groups related demonstrations rather than implying they call one another. Sampled sources show specific helper and streaming relationships; other capability links are based on the README’s package descriptions.
Main entry points: DatasetConversion.scala DataFrame Conversion · CustomReceiver.scala Custom Receiver · Ex1_SimpleRDD.scala RDD Examples · special Special RDDs
Components 19
RDD Processing
- RDD Examples
src/main/scala/Ex1_SimpleRDD.scala - Special RDDs
src/main/scala/special - Pair RDD
src/main/scala/special/PairRDD.scala
Structured Data
- DataFrame Conversion
src/main/scala/dataframe/DatasetConversion.scala - Rows And Schema
src/main/scala/dataframe/FromRowsAndSchema.scala - DataFrame Selection
src/main/scala/dataframe/Select.scala - Complex Datasets
src/main/scala/dataset/ComplexType.scala - Dataset Partitioning
src/main/scala/dataset/PartitionBy.scala - Table Hierarchy Utility
src/main/scala/utils/PartitionedTableHierarchy.scala
SQL And Hive
- SQL Examples
src/main/scala/sql - Hive Partitioned Tables
src/main/scala/hiveql/PartitionedTable.scala - Hive Sales UDAF
src/main/java/hiveql/SumLargeSalesUDAF.java
Streaming
- Custom Receiver
src/main/scala/streaming/CustomReceiver.scala - Full Monitoring
src/main/scala/streaming/FullMonitoring.scala - Structured Aggregation
src/main/scala/streaming/structured/BasicAggregation.scala - CSV Stream Generator
src/main/scala/streaming/util/CSVFileStreamGenerator.scala
Graph Processing
- Second-Degree Neighbors
src/main/scala/graphx/SecondDegreeNeighbors.scala
Other components
- Learner
- Apache Spark
Connections 17
- Learner → RDD Examples: runs example
- Learner → Special RDDs: runs examples
- Learner → DataFrame Conversion: runs example
- Learner → SQL Examples: runs examples
- Learner → Custom Receiver: runs example
- Learner → Second-Degree Neighbors: runs example
- RDD Examples → Apache Spark: demonstrates
- Special RDDs → Apache Spark: demonstrates
- DataFrame Conversion → Apache Spark: demonstrates
- SQL Examples → Apache Spark: demonstrates
- Custom Receiver → Apache Spark: demonstrates
- Second-Degree Neighbors → Apache Spark: demonstrates
- DataFrame Conversion → DataFrame Selection: uses selection
src/main/scala/dataframe/DatasetConversion.scala - Dataset Partitioning → Table Hierarchy Utility: calls utility
src/main/scala/dataset/PartitionBy.scala - Hive Partitioned Tables → Table Hierarchy Utility: calls utility
src/main/scala/hiveql/PartitionedTable.scala - Full Monitoring → Custom Receiver: registers receiver
src/main/scala/streaming/FullMonitoring.scala - Structured Aggregation → CSV Stream Generator: creates files
src/main/scala/streaming/structured/BasicAggregation.scala