# twitter/scalding architecture

> Architecture diagram of the GitHub repository twitter/scalding, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/twitter/scalding
- Repository: https://github.com/twitter/scalding

## Overview

Scalding is a Scala DSL for defining Hadoop MapReduce jobs. The documented user workflow reads typed input, transforms and groups records with Scala operations, then writes results; execution planning registers typed writes and connects a flow through the selected mode. The tree also shows alternate execution backends, data-format integrations, matrix utilities, and job-history estimators. Edges reflect the README workflow and sampled execution-context wiring; other subsystem links are limited where source excerpts were not provided.

Main entry points: [`scalding-core/src/main/scala/com/twitter/scalding/ExecutionContext.scala`](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/ExecutionContext.scala) (Execution Context), [`scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSource.scala`](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSource.scala) (Typed Sources), [`scalding-core/src/main/scala/com/twitter/scalding/Job.scala`](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/Job.scala) (Job), [`scalding-base/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala`](https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala) (TypedPipe DSL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Job API"]
  node_job["Job<br/>[Job.scala]"]
  node_args["Job Arguments<br/>[Args.scala]"]
end

subgraph group_data["Data Processing"]
  node_source["Typed Sources<br/>[TypedSource.scala]"]
  node_pipe["TypedPipe DSL<br/>[TypedPipe.scala]"]
  node_grouped["Grouping And Joins<br/>[Grouped.scala]"]
  node_matrix["Matrix Operations<br/>[Matrix.scala]"]
  node_sink["Typed Sinks<br/>[TypedSink.scala]"]
end

subgraph group_runtime["Execution Runtime"]
  node_execctx["Execution Context"]
  node_execution["Execution<br/>[Execution.scala]"]
  node_cascading["Cascading Backend"]
  node_beam["Beam Backend<br/>[BeamBackend.scala]"]
  node_memory["Memory Backend"]
end

subgraph group_integrations["Sources And Extensions"]
  node_files["Text And Sequence<br/>[TypedText.scala]"]
  node_avro["Avro Sources<br/>[AvroSource.scala]"]
  node_parquet["Parquet Sources"]
  node_commons["Lzo And Time Sources<br/>[DailySources.scala]"]
  node_estimation["Job Estimators<br/>[Estimator.scala]"]
  node_history["HRaven History"]
end

node_developer(("Job Author"))
node_hadoop["Hadoop And Cascading"]

node_developer -->|"defines job"| node_job
node_developer -->|"provides arguments"| node_args
node_job -->|"reads input"| node_source
node_job -->|"builds transformations"| node_pipe
node_pipe -->|"groups and joins"| node_grouped
node_job -->|"writes results"| node_sink
node_source -->|"supplies records"| node_pipe
node_pipe -->|"produces records"| node_sink
node_job -->|"executes plan"| node_execctx
node_execctx -->|"plans typed writes"| node_cascading
node_execctx -->|"connects flow"| node_hadoop
node_execctx -->|"runs flow"| node_execution
node_cascading -->|"builds flow"| node_hadoop
node_files -->|"provides file formats"| node_source
node_avro -->|"provides format"| node_source
node_parquet -->|"provides format"| node_source
node_commons -->|"provides sources"| node_source
node_pipe -->|"supports matrix work"| node_matrix
node_execctx -.->|"selects backend"| node_beam
node_execctx -.->|"selects backend"| node_memory
node_estimation -.->|"uses job history"| node_history

click node_job "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/Job.scala"
click node_args "https://github.com/twitter/scalding/blob/develop/scalding-args/src/main/scala/com/twitter/scalding/Args.scala"
click node_source "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSource.scala"
click node_pipe "https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala"
click node_grouped "https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/Grouped.scala"
click node_matrix "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/mathematics/Matrix.scala"
click node_sink "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSink.scala"
click node_execctx "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/ExecutionContext.scala"
click node_execution "https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/Execution.scala"
click node_cascading "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/cascading_backend/CascadingBackend.scala"
click node_beam "https://github.com/twitter/scalding/blob/develop/scalding-beam/src/main/scala/com/twitter/scalding/beam_backend/BeamBackend.scala"
click node_memory "https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/memory_backend/MemoryBackend.scala"
click node_files "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/source/TypedText.scala"
click node_avro "https://github.com/twitter/scalding/blob/develop/scalding-avro/src/main/scala/com/twitter/scalding/avro/AvroSource.scala"
click node_parquet "https://github.com/twitter/scalding/blob/develop/scalding-parquet/src/main/scala/com/twitter/scalding/parquet/HasColumnProjection.scala"
click node_commons "https://github.com/twitter/scalding/blob/develop/scalding-commons/src/main/scala/com/twitter/scalding/commons/source/DailySources.scala"
click node_estimation "https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/estimation/Estimator.scala"
click node_history "https://github.com/twitter/scalding/blob/develop/scalding-hraven/src/main/scala/com/twitter/scalding/hraven/estimation/HRavenHistoryService.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_job,node_args toneBlue
class node_source,node_pipe,node_grouped,node_matrix,node_sink toneAmber
class node_execctx,node_execution,node_cascading,node_beam,node_memory toneMint
class node_files,node_avro,node_parquet,node_commons,node_estimation,node_history toneRose
class node_developer,node_hadoop toneIndigo
```

## Components

### Job API

- [Job](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/Job.scala): `scalding-core/src/main/scala/com/twitter/scalding/Job.scala`
- [Job Arguments](https://github.com/twitter/scalding/blob/develop/scalding-args/src/main/scala/com/twitter/scalding/Args.scala): `scalding-args/src/main/scala/com/twitter/scalding/Args.scala`

### Data Processing

- [Typed Sources](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSource.scala): `scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSource.scala`
- [TypedPipe DSL](https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala): `scalding-base/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala`
- [Grouping And Joins](https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/Grouped.scala): `scalding-base/src/main/scala/com/twitter/scalding/typed/Grouped.scala`
- [Matrix Operations](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/mathematics/Matrix.scala): `scalding-core/src/main/scala/com/twitter/scalding/mathematics/Matrix.scala`
- [Typed Sinks](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSink.scala): `scalding-core/src/main/scala/com/twitter/scalding/typed/TypedSink.scala`

### Execution Runtime

- [Execution Context](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/ExecutionContext.scala): `scalding-core/src/main/scala/com/twitter/scalding/ExecutionContext.scala`
- [Execution](https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/Execution.scala): `scalding-base/src/main/scala/com/twitter/scalding/Execution.scala`
- [Cascading Backend](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/cascading_backend/CascadingBackend.scala): `scalding-core/src/main/scala/com/twitter/scalding/typed/cascading_backend/CascadingBackend.scala`
- [Beam Backend](https://github.com/twitter/scalding/blob/develop/scalding-beam/src/main/scala/com/twitter/scalding/beam_backend/BeamBackend.scala): `scalding-beam/src/main/scala/com/twitter/scalding/beam_backend/BeamBackend.scala`
- [Memory Backend](https://github.com/twitter/scalding/blob/develop/scalding-base/src/main/scala/com/twitter/scalding/typed/memory_backend/MemoryBackend.scala): `scalding-base/src/main/scala/com/twitter/scalding/typed/memory_backend/MemoryBackend.scala`

### Sources And Extensions

- [Text And Sequence](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/source/TypedText.scala): `scalding-core/src/main/scala/com/twitter/scalding/source/TypedText.scala`
- [Avro Sources](https://github.com/twitter/scalding/blob/develop/scalding-avro/src/main/scala/com/twitter/scalding/avro/AvroSource.scala): `scalding-avro/src/main/scala/com/twitter/scalding/avro/AvroSource.scala`
- [Parquet Sources](https://github.com/twitter/scalding/blob/develop/scalding-parquet/src/main/scala/com/twitter/scalding/parquet/HasColumnProjection.scala): `scalding-parquet/src/main/scala/com/twitter/scalding/parquet/HasColumnProjection.scala`
- [Lzo And Time Sources](https://github.com/twitter/scalding/blob/develop/scalding-commons/src/main/scala/com/twitter/scalding/commons/source/DailySources.scala): `scalding-commons/src/main/scala/com/twitter/scalding/commons/source/DailySources.scala`
- [Job Estimators](https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/estimation/Estimator.scala): `scalding-core/src/main/scala/com/twitter/scalding/estimation/Estimator.scala`
- [HRaven History](https://github.com/twitter/scalding/blob/develop/scalding-hraven/src/main/scala/com/twitter/scalding/hraven/estimation/HRavenHistoryService.scala): `scalding-hraven/src/main/scala/com/twitter/scalding/hraven/estimation/HRavenHistoryService.scala`

### Other components

- **Job Author**
- **Hadoop And Cascading**

## Connections

- Job Author → Job: defines job
- Job Author → Job Arguments: provides arguments
- Job → Typed Sources: reads input
- Job → TypedPipe DSL: builds transformations
- TypedPipe DSL → Grouping And Joins: groups and joins
- Job → Typed Sinks: writes results
- Typed Sources → TypedPipe DSL: supplies records
- TypedPipe DSL → Typed Sinks: produces records
- Job → Execution Context: executes plan
- Execution Context → Cascading Backend: plans typed writes
- Execution Context → Hadoop And Cascading: connects flow
- Execution Context → Execution: runs flow
- Cascading Backend → Hadoop And Cascading: builds flow
- Text And Sequence → Typed Sources: provides file formats
- Avro Sources → Typed Sources: provides format
- Parquet Sources → Typed Sources: provides format
- Lzo And Time Sources → Typed Sources: provides sources
- TypedPipe DSL → Matrix Operations: supports matrix work
- Execution Context → Beam Backend: selects backend
- Execution Context → Memory Backend: selects backend
- Job Estimators → HRaven History: uses job history