# sanket2905/data-engineering-lab architecture

> Architecture diagram of the GitHub repository sanket2905/data-engineering-lab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanket2905/data-engineering-lab
- Repository: https://github.com/sanket2905/data-engineering-lab

## Overview

Data Engineering Lab is a portfolio; the only documented implementation is a PySpark exercise for calculating insurance claim amounts from nested JSON. Its documented workflow reads claim data, flattens nested structures, explodes arrays, calculates treatment and medicine costs, aggregates deductions, and derives the final amount. The notebook and project-specific instructions were not provided as source excerpts, so these stages share the notebook path and the links are supported by the README rather than inspected implementation details.

Main entry points: [`projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb) (PySpark notebook), [`projects/pyspark/nested-json-insurance-claims/data/insurance_claim.json`](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/data/insurance_claim.json) (Claim JSON)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Claim analysis"]
  node_notebook["PySpark notebook"]
  node_flatten["Flatten nested data"]
  node_explode["Explode arrays"]
  node_costs["Calculate costs"]
  node_deductions["Aggregate deductions"]
  node_amount["Final claim amount"]
end

subgraph group_input["Claim input"]
  node_json["Claim JSON"]
end

node_engineer(("User"))

node_engineer -.->|"runs"| node_notebook
node_json -->|"provides input"| node_notebook
node_notebook -->|"flattens"| node_flatten
node_notebook -->|"explodes"| node_explode
node_notebook -->|"calculates costs"| node_costs
node_notebook -->|"aggregates"| node_deductions
node_notebook -->|"derives amount"| node_amount

click node_notebook "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb"
click node_json "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/data/insurance_claim.json"
click node_flatten "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb"
click node_explode "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb"
click node_costs "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb"
click node_deductions "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb"
click node_amount "https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_notebook,node_flatten,node_explode,node_costs,node_deductions,node_amount,node_engineer toneBlue
class node_json toneAmber
```

## Components

### Claim analysis

- [PySpark notebook](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb): `projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`
- [Flatten nested data](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb): `projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`
- [Explode arrays](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb): `projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`
- [Calculate costs](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb): `projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`
- [Aggregate deductions](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb): `projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`
- [Final claim amount](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb): `projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb`

### Claim input

- [Claim JSON](https://github.com/sanket2905/data-engineering-lab/blob/main/projects/pyspark/nested-json-insurance-claims/data/insurance_claim.json): `projects/pyspark/nested-json-insurance-claims/data/insurance_claim.json`

### Other components

- **User**

## Connections

- User → PySpark notebook: runs
- Claim JSON → PySpark notebook: provides input (evidence: `README.md`)
- PySpark notebook → Flatten nested data: flattens (evidence: `README.md`)
- PySpark notebook → Explode arrays: explodes (evidence: `README.md`)
- PySpark notebook → Calculate costs: calculates costs (evidence: `README.md`)
- PySpark notebook → Aggregate deductions: aggregates (evidence: `README.md`)
- PySpark notebook → Final claim amount: derives amount (evidence: `README.md`)
