# amazon-archives/emr-sample-apps architecture

> Architecture diagram of the GitHub repository amazon-archives/emr-sample-apps, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amazon-archives/emr-sample-apps
- Repository: https://github.com/amazon-archives/emr-sample-apps

## Overview

##### Project type and purpose
- This is a “sample code” repository illustrating how to run different data-processing frameworks and tools on Amazon EMR.
- It contains multiple independent modules (cloudburst, cloudfront, freebase, similarity, etc.), each demonstrating a particular EMR use case (Java MapReduce, Cascading, Python/MRJob, Pig, Hive-Ads, Spark/Shark, Node.js MR, encryption plugin for EMRFS, etc.).

##### File structure
- Top-level directories each represent an independent sample application or plugin.
- cloudburst: Java-based MapReduce alignment tool
- cloudfront: Java/Cascading log processor
- emrfs-plugins: EMRFS encryption provider
- freebase, similarity: Python/Ruby MRJob or Hadoop streaming jobs
- hive-ads, pig-apache: Hive and Pig scripts
- spark: Spark/Shark query samples
- node, wordcount: Hadoop streaming with Node.js / Python
- powershell: Windows automation scripts for Hive jobs
- No single interdependency across modules—they share the common target environment (Amazon EMR + S3 + SimpleDB).

##### README for additional insights
- The README states: “code samples … illustrate how to use popular applications on Amazon EMR.”

##### Detail without overthinking
- Logical EMR component and its responsibility.
- That all “sample modules” are just different job steps in the same overarching architecture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client and External
    Client["Client / CLI / SDK / PowerShell"]:::client
    S3[(Amazon S3)]:::storage
    SimpleDB[(Amazon SimpleDB)]:::storage

    %% EMR Cluster
    subgraph "EMR Cluster"
        direction TB
        Master["EMR Master Node"]:::compute
        Core["EMR Core Nodes"]:::compute
        EMRFS["EMRFS\n(with encryption plugin)"]:::plugin

        subgraph "Job Steps"
            direction TB
            CloudBurst["Java MapReduce\n(CloudBurst)"]:::jobstep
            click CloudBurst "https://github.com/amazon-archives/emr-sample-apps/blob/master/cloudburst/src/cloudBurst/CloudBurst.java"
            CloudFront["Java Cascading\n(CloudFront)"]:::jobstep
            click CloudFront "https://github.com/amazon-archives/emr-sample-apps/blob/master/cloudfront/code/src/logprocessor/Main.java"
            EMRFSPlugin["EMRFS Encryption\nPlugin"]:::jobstep
            click EMRFSPlugin "https://github.com/amazon-archives/emr-sample-apps/blob/master/emrfs-plugins/EMRFSRSAEncryptionMaterialsProvider/src/main/java/com/amazon/ws/emr/hadoop/fs/cse/RSAEncryptionMaterialsProvider.java"
            Freebase["Freebase MRJob\n(Python/Ruby)"]:::jobstep
            click Freebase "https://github.com/amazon-archives/emr-sample-apps/blob/master/freebase/code/mapper.py"
            click Freebase "https://github.com/amazon-archives/emr-sample-apps/blob/master/freebase/code/name_mapper.rb"
            click Freebase "https://github.com/amazon-archives/emr-sample-apps/blob/master/freebase/code/name_reducer.rb"
            click Freebase "https://github.com/amazon-archives/emr-sample-apps/blob/master/freebase/code/top_sdb_mapper.rb"
            click Freebase "https://github.com/amazon-archives/emr-sample-apps/blob/master/freebase/code/top_sdb_reducer.rb"
            click Freebase "https://github.com/amazon-archives/emr-sample-apps/blob/master/freebase/code/freebase_jobflow.json"
            Similarity["Similarity MRJob\n(Python)"]:::jobstep
            click Similarity "https://github.com/amazon-archives/emr-sample-apps/blob/master/similarity/similarity.py"
            click Similarity "https://github.com/amazon-archives/emr-sample-apps/blob/master/similarity/convert_netflix.py"
            click Similarity "https://github.com/amazon-archives/emr-sample-apps/blob/master/similarity/user_count_mapper.py"
            click Similarity "https://github.com/amazon-archives/emr-sample-apps/blob/master/similarity/lastfm_jobflow.json"
            click Similarity "https://github.com/amazon-archives/emr-sample-apps/blob/master/similarity/netflix_jobflow.json"
            HiveAds["Hive Queries\n(Hive-Ads)"]:::jobstep
            click HiveAds "https://github.com/amazon-archives/emr-sample-apps/blob/master/hive-ads/libs/join-clicks-to-impressions.q"
            click HiveAds "https://github.com/amazon-archives/emr-sample-apps/blob/master/hive-ads/libs/model-build.q"
            click HiveAds "https://github.com/amazon-archives/emr-sample-apps/blob/master/hive-ads/libs/response-time-stats.q"
            click HiveAds "https://github.com/amazon-archives/emr-sample-apps/blob/master/hive-ads/libs/twitter-impressions.q"
            Pig["Pig Scripts\n(Pig-Apache)"]:::jobstep
            click Pig "https://github.com/amazon-archives/emr-sample-apps/blob/master/pig-apache/do-reports.pig"
            click Pig "https://github.com/amazon-archives/emr-sample-apps/blob/master/pig-apache/do-reports2.pig"
            Spark["Spark/Shark\nQueries (spark)"]:::jobstep
            click Spark "https://github.com/amazon-archives/emr-sample-apps/blob/master/spark/Spark sample query.txt"
            click Spark "https://github.com/amazon-archives/emr-sample-apps/blob/master/spark/Shark sample query.txt"
            Node["Node.js Streaming\n(WordCount)"]:::jobstep
            click Node "https://github.com/amazon-archives/emr-sample-apps/blob/master/node/sample-mapper.js"
            click Node "https://github.com/amazon-archives/emr-sample-apps/blob/master/node/sample-reducer.js"
            WordCount["Python Streaming\n(WordCount)"]:::jobstep
            click WordCount "https://github.com/amazon-archives/emr-sample-apps/blob/master/wordcount/wordSplitter.py"
            PowerShell["PowerShell\nAutomation"]:::jobstep
            click PowerShell "https://github.com/amazon-archives/emr-sample-apps/blob/master/powershell/Create-InteractiveHiveJob.ps1"
            click PowerShell "https://github.com/amazon-archives/emr-sample-apps/blob/master/powershell/RunHiveDemoJob.ps1"
        end
    end

    %% Relationships
    Client -->|"submit job flow"| Master
    Client -->|"launch via PowerShell"| Master
    Master -->|"fetch code & data"| S3
    Master -->|"distribute steps"| Core
    Core -->|"execute steps"| Job_Steps((Job Steps))
    Core -->|"read/write data"| S3
    EMRFS -->|"read/write data"| S3
    Core -->|"uses EMRFS"| EMRFS
    Core -->|"persist/query"| SimpleDB
    Master -->|"status & logs"| Client
    Master -->|"store logs & output"| S3

    %% Classes
    classDef storage fill:#a3d5ff,stroke:#036,stroke-width:1px;
    classDef compute fill:#ffd699,stroke:#963,stroke-width:1px;
    classDef client fill:#b3ffb3,stroke:#070,stroke-width:1px;
    classDef plugin fill:#ffcccc,stroke:#933,stroke-width:1px;
    classDef jobstep fill:#eeeeee,stroke:#999,stroke-width:1px;
```
