How twitter/scalding works
Scalding is a Scala DSL for defining Hadoop MapReduce jobs. The documented user workflow reads typed input, transforms and groups records with Scala operations, then writes results; execution planning registers typed writes and connects a flow through the selected mode. The tree also shows alternate execution backends, data-format integrations, matrix utilities, and job-history estimators. Edges reflect the README workflow and sampled execution-context wiring; other subsystem links are limited where source excerpts were not provided.
Main entry points: ExecutionContext.scala Execution Context · TypedSource.scala Typed Sources · Job.scala Job · TypedPipe.scala TypedPipe DSL
Components 20
Job API
- Job
scalding-core/src/main/scala/com/twitter/scalding/Job.scala - Job Arguments
scalding-args/src/main/scala/com/twitter/scalding/Args.scala
Data Processing
- Typed Sources
scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSource.scala - TypedPipe DSL
scalding-base/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala - Grouping And Joins
scalding-base/src/main/scala/com/twitter/scalding/typed/Grouped.scala - Matrix Operations
scalding-core/src/main/scala/com/twitter/scalding/mathematics/Matrix.scala - Typed Sinks
scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSink.scala
Execution Runtime
- Execution Context
scalding-core/src/main/scala/com/twitter/scalding/ExecutionContext.scala - Execution
scalding-base/src/main/scala/com/twitter/scalding/Execution.scala - Cascading Backend
scalding-core/src/main/scala/com/twitter/scalding/typed/cascading_backend/CascadingBackend.scala - Beam Backend
scalding-beam/src/main/scala/com/twitter/scalding/beam_backend/BeamBackend.scala - Memory Backend
scalding-base/src/main/scala/com/twitter/scalding/typed/memory_backend/MemoryBackend.scala
Sources And Extensions
- Text And Sequence
scalding-core/src/main/scala/com/twitter/scalding/source/TypedText.scala - Avro Sources
scalding-avro/src/main/scala/com/twitter/scalding/avro/AvroSource.scala - Parquet Sources
scalding-parquet/src/main/scala/com/twitter/scalding/parquet/HasColumnProjection.scala - Lzo And Time Sources
scalding-commons/src/main/scala/com/twitter/scalding/commons/source/DailySources.scala - Job Estimators
scalding-core/src/main/scala/com/twitter/scalding/estimation/Estimator.scala - HRaven History
scalding-hraven/src/main/scala/com/twitter/scalding/hraven/estimation/HRavenHistoryService.scala
Other components
- Job Author
- Hadoop And Cascading
Connections 21
- Job Author → Job: defines job
- Job Author → Job Arguments: provides arguments
- Job → Typed Sources: reads input
- Job → TypedPipe DSL: builds transformations
- TypedPipe DSL → Grouping And Joins: groups and joins
- Job → Typed Sinks: writes results
- Typed Sources → TypedPipe DSL: supplies records
- TypedPipe DSL → Typed Sinks: produces records
- Job → Execution Context: executes plan
- Execution Context → Cascading Backend: plans typed writes
- Execution Context → Hadoop And Cascading: connects flow
- Execution Context → Execution: runs flow
- Cascading Backend → Hadoop And Cascading: builds flow
- Text And Sequence → Typed Sources: provides file formats
- Avro Sources → Typed Sources: provides format
- Parquet Sources → Typed Sources: provides format
- Lzo And Time Sources → Typed Sources: provides sources
- TypedPipe DSL → Matrix Operations: supports matrix work
- Execution Context → Beam Backend: selects backend
- Execution Context → Memory Backend: selects backend
- Job Estimators → HRaven History: uses job history