How sahilpatil2103/aws-spark-streaming-on-unstructured-data works
Project type and purpose
• This is a real-time big-data processing pipeline built on Apache Spark Streaming, containerized via Docker, for extracting structured insights from unstructured job-posting data.
• It reads text/JSON job postings, applies Python UDFs, writes the cleaned/structured results to AWS S3, and leverages AWS Glue Crawler and Athena (Athenna) for downstream analysis.
File structure
• Top-level folders:
• Datasets – raw input files (ACCOUNTING CLERK.txt, JobPostings.json)
• Docker – DockerFile and docker-compose.yml to spin up a Spark master + workers cluster
• Jobs – Spark application code:
– Config/config.py (connection settings, S3 paths, streaming configs)
– main.py (Spark Streaming driver)
– udf_utils.py (custom parsing/cleaning functions)
• Execution scripts: EXECUTION_GUIDE.md
• Build/deployment artifacts: DockerFile, docker-compose.yml
README for architectural insights
• Confirms Docker-based Spark cluster (Bitnami Spark image), Python3.9/Java11, AWS S3 as sink
• Mentions AWS IAM for permissions, Glue Crawler for schema discovery, and Athenna (Athena) for query access
a. Main components
1. Data Source Layer
– Raw files in Datasets folder or real-time ingestion endpoint
2. Containerized Spark Cluster
– spark-master (driver & web UI)
– spark-worker nodes (executors)
– Defined and orchestrated by docker-compose.yml
3. Streaming Application
– Jobs/main.py submits Spark Streaming job
– Config module loads parameters (S3 bucket, streaming batch interval)
– udf_utils parses unstructured text/JSON
4. Cloud Storage & Catalog
– AWS S3 bucket (structured output, partitioned)
– AWS Glue Crawler (discovers schema)
5. Query Layer
– AWS Athena (Athenna) queries on S3
6. IAM & Security
– IAM roles/policies granting Spark containers S3 and Glue access
b. Relationships & Data Flow
• Spark Streaming jobs communicate with Config to get S3 endpoints
• Processed Data → AWS S3 (s3a://…)
• AWS Glue Crawler polls S3 → updates Data Catalog
• AWS Athena queries cataloged tables
• Docker Compose orchestrates Spark master ↔ workers communication
c. Architectural Patterns & Principles
• Micro-service-style containerization for scalability
• Lambda-style streaming (real-time) processing
• Separation of concerns: ingestion, processing, storage, analytics
• Infrastructure as code via Docker Compose
d. Key Technologies
• Apache Spark (PySpark streaming)
• Docker / Docker Compose (Bitnami Spark image)
• Python 3.9, Java 11
• AWS S3, AWS Glue Crawler, AWS Athena (Athenna)
• Custom Python UDFs for parsing