How sanket2905/data-engineering-lab works
Data Engineering Lab is a portfolio; the only documented implementation is a PySpark exercise for calculating insurance claim amounts from nested JSON. Its documented workflow reads claim data, flattens nested structures, explodes arrays, calculates treatment and medicine costs, aggregates deductions, and derives the final amount. The notebook and project-specific instructions were not provided as source excerpts, so these stages share the notebook path and the links are supported by the README rather than inspected implementation details.
Main entry points: insurance_claims_analysis.ipynb PySpark notebook · insurance_claim.json Claim JSON
Components 8
Claim analysis
- PySpark notebook
projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb - Flatten nested data
projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb - Explode arrays
projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb - Calculate costs
projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb - Aggregate deductions
projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb - Final claim amount
projects/pyspark/nested-json-insurance-claims/insurance_claims_analysis.ipynb
Claim input
- Claim JSON
projects/pyspark/nested-json-insurance-claims/data/insurance_claim.json
Other components
- User
Connections 7
- User → PySpark notebook: runs
- Claim JSON → PySpark notebook: provides input
README.md - PySpark notebook → Flatten nested data: flattens
README.md - PySpark notebook → Explode arrays: explodes
README.md - PySpark notebook → Calculate costs: calculates costs
README.md - PySpark notebook → Aggregate deductions: aggregates
README.md - PySpark notebook → Final claim amount: derives amount
README.md