How mironathome/pyspark-analyzer works
Project Type and Purpose
• This is an open-source Python library (“PyPI package”) providing a comprehensive PySpark DataFrame profiling tool.
• It exposes a single, unified API function (analyze(df, …)) which, under the hood, orchestrates sampling, caching, aggregation, and statistics computation on a Spark DataFrame and returns a summary in pandas, dict, or summary text formats.
High-Level Architecture & File-Structure Analysis
• Top-level directories:
• pyspark_analyzer/ – core library modules (sampling, statistics, caching, profiling, utils, logging, performance, quality, progress).
• examples/ – usage scripts demonstrating basic to advanced features.
• tests/ – pytest suite validating each component.
• docs/ – Sphinx-based documentation.
• scripts/ – helper scripts for CI/test environment.
• Config & CI/CD:
• .github/workflows/ – builds, tests, docs, security scans.
• Makefile, pyproject.toml, pre-commit-config.yaml – build/test automation.
Core Components & Their Interactions
a. External Inputs
• User code / Notebook / CLI calls analyze(df, config…).
• Downstream: a SparkSession and distributed DataFrame on a Spark cluster.
b. Front Door / API Layer
• api.py defines analyze() entrypoint.
• Parses user options into a SamplingConfig (in caching.py & constants.py).
c. Sampling & Partitioning
• SamplingConfig decides: full scan vs. sampling (auto / fraction / fixed size).
• adaptive_partitioning logic (in performance.py) optimizes number of partitions using cluster config and AQE.
d. Statistics Computation
• BatchStatisticsComputer orchestrates a single, combined pass over the DataFrame.
• For each column type, delegates to StatisticsComputer (in statistics.py) and specialized submodules (quality.py, profiler.py, sampling.py, progress.py).
• Uses approximate functions (distinct counts, percentiles) and single-scan aggregations for speed.
e. Caching & Performance
• Intelligent caching (in caching.py) holds intermediate results in memory across multiple computations.
• Logging and progress reporting via logging.py and progress.py.
f. Output Formatting
• Converts Spark-aggregated results into pandas DataFrame, dict, or human-readable summary (in utils.py).
• Attaches metadata (overview, sampling) via pandas attrs or dictionary keys.
Essential Elements without Over-Modularizing
• Internal Python modules into logical sub-systems rather than showing every .py file
• How analyze() orchestrates the workflow
• Where sampling/partitioning decisions occur
• How Spark executes the heavy lifting
• How results are formatted and returned