# mironathome/pyspark-analyzer architecture

> Architecture diagram of the GitHub repository mironathome/pyspark-analyzer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mironathome/pyspark-analyzer
- Repository: https://github.com/mironathome/pyspark-analyzer

## Overview

##### Project Type and Purpose
- This is an open-source Python library (“PyPI package”) providing a comprehensive PySpark DataFrame profiling tool.
- It exposes a single, unified API function (`analyze(df, …)`) which, under the hood, orchestrates sampling, caching, aggregation, and statistics computation on a Spark DataFrame and returns a summary in pandas, dict, or summary text formats.

##### High-Level Architecture & File-Structure Analysis
- Top-level directories:
- `pyspark_analyzer/` – core library modules (sampling, statistics, caching, profiling, utils, logging, performance, quality, progress).
- `examples/` – usage scripts demonstrating basic to advanced features.
- `tests/` – pytest suite validating each component.
- `docs/` – Sphinx-based documentation.
- `scripts/` – helper scripts for CI/test environment.
- Config & CI/CD:
- `.github/workflows/` – builds, tests, docs, security scans.
- `Makefile`, `pyproject.toml`, `pre-commit-config.yaml` – build/test automation.

##### Core Components & Their Interactions
a. External Inputs
- User code / Notebook / CLI calls `analyze(df, config…)`.
- Downstream: a SparkSession and distributed DataFrame on a Spark cluster.
b. Front Door / API Layer
- `api.py` defines `analyze()` entrypoint.
- Parses user options into a `SamplingConfig` (in `caching.py` & `constants.py`).
c. Sampling & Partitioning
- `SamplingConfig` decides: full scan vs. sampling (auto / fraction / fixed size).
- `adaptive_partitioning` logic (in `performance.py`) optimizes number of partitions using cluster config and AQE.
d. Statistics Computation
- `BatchStatisticsComputer` orchestrates a single, combined pass over the DataFrame.
- For each column type, delegates to `StatisticsComputer` (in `statistics.py`) and specialized submodules (`quality.py`, `profiler.py`, `sampling.py`, `progress.py`).
- Uses approximate functions (distinct counts, percentiles) and single-scan aggregations for speed.
e. Caching & Performance
- Intelligent caching (in `caching.py`) holds intermediate results in memory across multiple computations.
- Logging and progress reporting via `logging.py` and `progress.py`.
f. Output Formatting
- Converts Spark-aggregated results into pandas DataFrame, dict, or human-readable summary (in `utils.py`).
- Attaches metadata (`overview`, `sampling`) via pandas attrs or dictionary keys.

##### Essential Elements without Over-Modularizing
- Internal Python modules into logical sub-systems rather than showing every `.py` file
- How `analyze()` orchestrates the workflow
- Where sampling/partitioning decisions occur
- How Spark executes the heavy lifting
- How results are formatted and returned

## Diagram (Mermaid)

```mermaid
flowchart TB
    A["User (Notebook/Script)"]:::external
    subgraph "PySpark Analyzer"
        API["API: analyze()"]:::api
        CONF["Configuration: SamplingConfig"]:::api
        SAM["Sampling Engine"]:::sampling
        PART["Adaptive Partitioner"]:::sampling
        STAT["Statistics Engine"]:::stats
        PROF["Profiler Modules"]:::stats
        QUAL["Quality Modules"]:::stats
        CACHE["Caching Layer"]:::caching
        LOG["Logging"]:::caching
        PROG["Progress Reporting"]:::caching
        EXC["Exceptions"]:::caching
        FORM["Output Formatter"]:::formatting
    end
    SPARK["Spark Cluster\n(Driver + Executors)"]:::external
    DS["Data Sources\n(Parquet/CSV)"]:::external
    OUT["Formatted Output\n(pandas/dict/text)"]:::external

    A -->|"calls"| API
    API -->|"parses options"| CONF
    CONF -->|"sampling decision"| SAM
    SAM -->|"partition info"| PART
    PART -->|"transforms DF"| SPARK
    CONF -->|"cache settings"| CACHE
    API -->|"start batch"| STAT
    STAT -->|"delegates"| PROF
    STAT -->|"delegates"| QUAL
    CACHE -->|"caches RDDs"| SPARK
    SPARK -->|"executes jobs"| STAT
    SPARK -->|"reads data"| DS
    STAT -->|"results"| FORM
    FORM -->|"returns output"| A
    LOG -->|"logs"| PROG

    click API "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/api.py"
    click CONF "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/constants.py"
    click SAM "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/sampling.py"
    click PART "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/performance.py"
    click STAT "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/statistics.py"
    click PROF "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/profiler.py"
    click QUAL "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/quality.py"
    click CACHE "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/caching.py"
    click LOG "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/logging.py"
    click PROG "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/progress.py"
    click EXC "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/exceptions.py"
    click FORM "https://github.com/mironathome/pyspark-analyzer/blob/main/pyspark_analyzer/utils.py"

    classDef api fill:#D0E8FF,stroke:#0E64BC
    classDef sampling fill:#E4F7D1,stroke:#3C8A0E
    classDef stats fill:#FFE5B4,stroke:#D35400
    classDef caching fill:#F3E6FF,stroke:#7D3C98
    classDef formatting fill:#FFF2CC,stroke:#B97A00
    classDef external fill:#DDDDDD,stroke:#888888
```
