# neeleshsamptur/bd_project architecture

> Architecture diagram of the GitHub repository neeleshsamptur/bd_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neeleshsamptur/bd_project
- Repository: https://github.com/neeleshsamptur/bd_project

## Overview

##### Project Type and Purpose
- This is a code-transformation tool (an open-source utility) that automates converting Spark batch jobs into their equivalent Spark streaming jobs.
- It uses the Python ast library to parse the batch code’s abstract syntax tree and emits a streaming version of that code.

##### File Structure Analysis
- Top-level scripts:
- ast_converter.py – the core engine that reads a batch script’s AST and generates streaming code
- batch.py – example batch-processing Spark job that reads CSV data and writes output
- stream.py – equivalent streaming job template that processes CSV streams
- Data directories:
- batch_input/ – contains source CSV for batch runs
- batch_output/ – holds batch job results (CSV)
- stream_input/ – simulates input streams (multiple CSV files)
- No explicit stream_output folder, implying streaming job writes to console or is instrumented elsewhere.

##### Key README Insights
- Dependencies: Python, pyspark (for Spark jobs), ast (for tree parsing/manipulation)
- Goal: “convert a batch code to its equivalent stream code”

##### Main Components
a. Data Sources
– batch_input (static CSV)
– stream_input (simulated streaming CSV files)
b. Batch Processing Application
– batch.py (runs Spark batch job via pyspark APIs)
– Writes results to batch_output
c. AST Conversion Engine
– ast_converter.py
– Parses batch.py AST → applies transformation rules → emits stream.py
d. Streaming Processing Application
– stream.py (runs Spark Structured Streaming via pyspark)
– Consumes stream_input as micro-batches or continuous stream
e. Dependencies / Infrastructure
– Python runtime
– pyspark library
– ast library

##### Architectural Patterns and Design Principles
- Separation of concerns:
- Data ingestion components (inputs)
- Processing logic (batch vs streaming scripts)
- Code-generation engine (AST converter)
- Code-transformation pipeline: parse → transform → emit
- Reuse of Spark APIs – only topologies differ (batch vs streaming)

##### Final Touches
- Number the processing stages: 1) Batch run, 2) AST transform, 3) Stream run.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "1. Data Sources"
        DataBatch["batch_input/"]:::dataStore
        DataStream["stream_input/"]:::dataStore
    end

    subgraph "2. Processing"
        BatchApp["batch.py"]:::processing
        ASTEngine["ast_converter.py"]:::engine
        StreamApp["stream.py"]:::processing
    end

    subgraph "3. Outputs"
        BatchOutput["batch_output/"]:::dataStore
        Console["Console Sink"]:::processing
    end

    subgraph "Infrastructure"
        Python["Python Runtime"]:::infra
        PySpark["pyspark library"]:::infra
        ASTLib["ast library"]:::infra
        Readme["README.md"]:::doc
    end

    DataBatch -->|"reads CSV"| BatchApp
    BatchApp -->|"writes CSV"| BatchOutput
    BatchApp -->|"source code"| ASTEngine
    ASTEngine -->|"emits code"| StreamApp
    DataStream -->|"streams CSV"| StreamApp
    StreamApp -->|"outputs stream"| Console

    BatchApp -.->|"depends on"| PySpark
    StreamApp -.->|"depends on"| PySpark
    ASTEngine -.->|"depends on"| ASTLib

    BatchApp -.->|"runs on"| Python
    StreamApp -.->|"runs on"| Python
    ASTEngine -.->|"runs on"| Python

    click DataBatch "https://github.com/neeleshsamptur/bd_project/tree/main/batch_input/"
    click DataStream "https://github.com/neeleshsamptur/bd_project/tree/main/stream_input/"
    click BatchApp "https://github.com/neeleshsamptur/bd_project/blob/main/batch.py"
    click BatchOutput "https://github.com/neeleshsamptur/bd_project/tree/main/batch_output/"
    click ASTEngine "https://github.com/neeleshsamptur/bd_project/blob/main/ast_converter.py"
    click StreamApp "https://github.com/neeleshsamptur/bd_project/blob/main/stream.py"
    click Readme "https://github.com/neeleshsamptur/bd_project/blob/main/README.md"

    classDef dataStore fill:#e0ffe0,stroke:#2f6627;
    classDef processing fill:#cce5ff,stroke:#2f4d7c;
    classDef engine fill:#ffe5cc,stroke:#b35900;
    classDef infra fill:#f0f0f0,stroke:#888888;
    classDef doc fill:#fff0f5,stroke:#d147a3;
```
