# sahilpatil2103/aws-spark-streaming-on-unstructured-data architecture

> Architecture diagram of the GitHub repository sahilpatil2103/aws-spark-streaming-on-unstructured-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data
- Repository: https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data

## Overview

##### Project type and purpose
- This is a real-time big-data processing pipeline built on Apache Spark Streaming, containerized via Docker, for extracting structured insights from unstructured job-posting data.
- It reads text/JSON job postings, applies Python UDFs, writes the cleaned/structured results to AWS S3, and leverages AWS Glue Crawler and Athena (Athenna) for downstream analysis.

##### File structure
- Top-level folders:
- Datasets – raw input files (ACCOUNTING CLERK.txt, JobPostings.json)
- Docker – DockerFile and docker-compose.yml to spin up a Spark master + workers cluster
- Jobs – Spark application code:
– Config/config.py (connection settings, S3 paths, streaming configs)
– main.py (Spark Streaming driver)
– udf_utils.py (custom parsing/cleaning functions)
- Execution scripts: EXECUTION_GUIDE.md
- Build/deployment artifacts: DockerFile, docker-compose.yml

##### README for architectural insights
- Confirms Docker-based Spark cluster (Bitnami Spark image), Python3.9/Java11, AWS S3 as sink
- Mentions AWS IAM for permissions, Glue Crawler for schema discovery, and Athenna (Athena) for query access

a. Main components
1. Data Source Layer
– Raw files in Datasets folder or real-time ingestion endpoint
2. Containerized Spark Cluster
– spark-master (driver & web UI)
– spark-worker nodes (executors)
– Defined and orchestrated by docker-compose.yml
3. Streaming Application
– Jobs/main.py submits Spark Streaming job
– Config module loads parameters (S3 bucket, streaming batch interval)
– udf_utils parses unstructured text/JSON
4. Cloud Storage & Catalog
– AWS S3 bucket (structured output, partitioned)
– AWS Glue Crawler (discovers schema)
5. Query Layer
– AWS Athena (Athenna) queries on S3
6. IAM & Security
– IAM roles/policies granting Spark containers S3 and Glue access

b. Relationships & Data Flow
- Spark Streaming jobs communicate with Config to get S3 endpoints
- Processed Data → AWS S3 (s3a://…)
- AWS Glue Crawler polls S3 → updates Data Catalog
- AWS Athena queries cataloged tables
- Docker Compose orchestrates Spark master ↔ workers communication

c. Architectural Patterns & Principles
- Micro-service-style containerization for scalability
- Lambda-style streaming (real-time) processing
- Separation of concerns: ingestion, processing, storage, analytics
- Infrastructure as code via Docker Compose

d. Key Technologies
- Apache Spark (PySpark streaming)
- Docker / Docker Compose (Bitnami Spark image)
- Python 3.9, Java 11
- AWS S3, AWS Glue Crawler, AWS Athena (Athenna)
- Custom Python UDFs for parsing

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Sources"
        direction TB
        File1["ACCOUNTING CLERK.txt"]:::ingest
        File2["JobPostings.json"]:::ingest
        File1 --> SparkMaster
        File2 --> SparkMaster
    end

    subgraph "On-Prem: Docker-based Spark Cluster"
        direction TB
        DockerCompose["docker-compose.yml"]:::processing
        DockerFile["DockerFile"]:::processing
        SparkMaster["Spark Master\n(driver & UI)"]:::processing
        SparkWorker["Spark Workers\n(executors)"]:::processing
        DockerCompose --> SparkMaster
        DockerCompose --> SparkWorker
    end

    subgraph "Streaming Application"
        direction TB
        Main["main.py\n(Spark Streaming Driver)"]:::processing
        Config["config.py\n(parameters & S3 paths)"]:::processing
        UDFs["udf_utils.py\n(custom parsing UDFs)"]:::processing
        Main --> Config
        Main --> UDFs
        SparkMaster --> Main
        SparkWorker --> Main
    end

    subgraph "Cloud Storage & Catalog"
        direction TB
        S3Bucket["AWS S3 Bucket"]:::storage
        GlueCrawler["AWS Glue Crawler"]:::storage
        DataCatalog["Glue Data Catalog"]:::storage
        Main -->|"s3a://write"| S3Bucket
        SparkWorker -->|"s3a://write"| S3Bucket
        S3Bucket --> GlueCrawler
        GlueCrawler --> DataCatalog
    end

    subgraph "Analytics"
        direction TB
        Athena["AWS Athena"]:::analytics
        DataCatalog --> Athena
    end

    subgraph "Security & IAM"
        direction TB
        IAM["AWS IAM Roles & Policies"]:::security
        SparkMaster --> IAM
        SparkWorker --> IAM
        GlueCrawler --> IAM
        Athena --> IAM
    end

    subgraph "Documentation & References"
        direction TB
        ExecGuide["EXECUTION_GUIDE.md"]:::doc
        DiagramRef["Architecture Diagram (reference)"]:::doc
        Readme["README.md"]:::doc
    end

    click File1 "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/Datasets/ACCOUNTING CLERK.txt"
    click File2 "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/Datasets/JobPostings.json"
    click DockerCompose "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/Docker/docker-compose.yml"
    click DockerFile "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/tree/main/Docker/DockerFile"
    click Main "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/Jobs/main.py"
    click Config "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/Jobs/Config/config.py"
    click UDFs "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/Jobs/udf_utils.py"
    click ExecGuide "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/EXECUTION_GUIDE.md"
    click DiagramRef "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/tree/main/Images/"
    click Readme "https://github.com/sahilpatil2103/aws-spark-streaming-on-unstructured-data/blob/main/README.md"

    classDef ingest fill:#B3DDF2,stroke:#333,stroke-width:1px
    classDef processing fill:#AED6F1,stroke:#333,stroke-width:1px
    classDef storage fill:#A9DFBF,stroke:#333,stroke-width:1px
    classDef analytics fill:#F5CBA7,stroke:#333,stroke-width:1px
    classDef security fill:#D2B4DE,stroke:#333,stroke-width:1px
    classDef doc fill:#E5E7E9,stroke:#333,stroke-width:1px
```
