How neeleshsamptur/bd_project works
Project Type and Purpose
• This is a code-transformation tool (an open-source utility) that automates converting Spark batch jobs into their equivalent Spark streaming jobs.
• It uses the Python ast library to parse the batch code’s abstract syntax tree and emits a streaming version of that code.
File Structure Analysis
• Top-level scripts:
• ast_converter.py – the core engine that reads a batch script’s AST and generates streaming code
• batch.py – example batch-processing Spark job that reads CSV data and writes output
• stream.py – equivalent streaming job template that processes CSV streams
• Data directories:
• batch_input/ – contains source CSV for batch runs
• batch_output/ – holds batch job results (CSV)
• stream_input/ – simulates input streams (multiple CSV files)
• No explicit stream_output folder, implying streaming job writes to console or is instrumented elsewhere.
Key README Insights
• Dependencies: Python, pyspark (for Spark jobs), ast (for tree parsing/manipulation)
• Goal: “convert a batch code to its equivalent stream code”
Main Components
a. Data Sources
– batch_input (static CSV)
– stream_input (simulated streaming CSV files)
b. Batch Processing Application
– batch.py (runs Spark batch job via pyspark APIs)
– Writes results to batch_output
c. AST Conversion Engine
– ast_converter.py
– Parses batch.py AST → applies transformation rules → emits stream.py
d. Streaming Processing Application
– stream.py (runs Spark Structured Streaming via pyspark)
– Consumes stream_input as micro-batches or continuous stream
e. Dependencies / Infrastructure
– Python runtime
– pyspark library
– ast library
Architectural Patterns and Design Principles
• Separation of concerns:
• Data ingestion components (inputs)
• Processing logic (batch vs streaming scripts)
• Code-generation engine (AST converter)
• Code-transformation pipeline: parse → transform → emit
• Reuse of Spark APIs – only topologies differ (batch vs streaming)
Final Touches
• Number the processing stages: 1) Batch run, 2) AST transform, 3) Stream run.