# mrpowers/chispa architecture

> Architecture diagram of the GitHub repository mrpowers/chispa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mrpowers/chispa
- Repository: https://github.com/mrpowers/chispa

## Overview

##### Project Type and Purpose
- chispa is a testing utility for PySpark. Its primary purpose is to provide helper methods (assertions) that allow users to verify column equality, DataFrame equality (both exact and approximate), schema comparisons, and various formatting options for error messages.
- Its design helps developers write high-quality PySpark code by generating descriptive and easy-to-read error messages when tests fail.
- The README clearly shows usage examples, including functions like assert_column_equality, assert_df_equality, assert_approx_df_equality, and format customization functions.

##### Analyzing the File Structure
- The package’s core code is contained in the “chispa” directory. Key modules include:
- __init__.py – The module initializer that likely exposes the main APIs.
- column_comparer.py – Contains the function assert_column_equality to compare individual columns.
- dataframe_comparer.py – Implements assert_df_equality, assert_approx_df_equality, and other functions for comparing entire DataFrames.
- formatting (directory) – Holds modules such as:
▸ format_string.py – Likely for generating formatted string representations.
▸ formats.py – Possibly defines available formatting styles.
▸ formatting_config.py – Defines the FormattingConfig class that allows users to customize output styles.
- number_helpers.py – Contains numeric helper functions possibly used in approximate equality comparisons.
- row_comparer.py and rows_comparer.py – Implement functions for comparing individual rows as well as lists of rows.
- schema_comparer.py and structfield_comparer.py – Contain utilities to compare schema definitions and properties (including nullability).
- The “tests” directory contains comprehensive unit tests that exercise each of these functions and classes, ensuring that comparison and formatting behave as expected.

##### Key Functions and Classes (the “functions and classes used”)
- assert_column_equality (in column_comparer.py) – Compares two columns in a DataFrame.
- assert_df_equality (in dataframe_comparer.py) – Compares two DataFrames, potentially with flags such as ignore_row_order, ignore_column_order, or ignore_nullable.
- assert_approx_column_equality and assert_approx_df_equality – Provide approximate equality checks for floating point values.
- Additional helpers include functions found in row_comparer.py, rows_comparer.py, schema_comparer.py, and structfield_comparer.py to break down the equality checks into granular parts.
- bcolors.py defines constants or utility functions for terminal coloration, useful across the comparers.
- The tests themselves (such as test_column_comparer, test_dataframe_comparer, test_row_comparer, etc.) illustrate the usage patterns of these functions and classes.

a. Main Components to Represent:
- chispa Core Package:
▸ Comparer Modules: Column Comparer, DataFrame Comparer, Row Comparer, Schema Comparer, StructField Comparer.
▸ Formatting Module: Formatting configuration, format string helpers, and default styles (formats.py, formatting_config.py).
▸ Utils: bcolors and number_helpers providing supportive functionality.
- External Dependencies:
▸ PySpark: The library’s API for DataFrame operations.
▸ Python’s standard libraries.
- Testing & CI Infrastructure:
▸ "tests" directory – Contains unit and integration tests.
▸ CI/CD setup on GitHub (.github/workflows etc.), config files (ci directory, Makefile, pyproject.toml), and documentation (docs, mkdocs.yml).

b. Relationships Between Components:
- The various comparer modules interact with PySpark by directly comparing DataFrame contents, schemas, and rows.
- FormattingConfig is used by comparers to present error messages in a user-friendly way, often calling formatting helpers in the “formatting” directory.
- The bcolors and number_helpers modules provide utility methods that feed into both the formatting and comparison logic.

c. Highlighting Architectural Patterns:
- The project uses a modular design where each area of functionality is separated into its own module (e.g., separating the logical concerns of column vs. DataFrame vs. row comparisons).
- This separation allows for clear dependency paths—for instance, formatting functions are decoupled from the comparison logic.
- The use of configuration (via FormattingConfig) allows for high customizability and reuse across the library’s functions.

##### Final Advice
- Key functions (like assert_column_equality, assert_df_equality, assert_approx_df_equality) and the main classes (like FormattingConfig) by placing them as annotations inside their corresponding component blocks.
- Revisiting the README and tests can help verify that all important functions and classes are appropriately represented.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dependencies
    subgraph "External Dependencies"
        EX1("PySpark"):::external
        EX2("Python StdLib"):::external
    end

    %% Chispa Core Library
    subgraph "Chispa Core Library"
        API("API Initialization"):::core

        subgraph "Comparers"
            CC("Column Comparer\n(assert_column_equality)"):::comparer
            DC("DataFrame Comparer\n(assert_df_equality,assert_approx_df_equality)"):::comparer
            RC("Row Comparer\n(row_comparer)"):::comparer
            RSC("Rows Comparer\n(rows_comparer)"):::comparer
            SC("Schema Comparer\n(schema_comparer)"):::comparer
            SFC("StructField Comparer\n(structfield_comparer)"):::comparer
        end

        subgraph "Formatting"
            FC("Formatting Config\n(FormattingConfig)"):::formatting
            FSH("Format String Helper\n(format_string)"):::formatting
            FD("Formats Definitions\n(formats)"):::formatting
            DF("Default Formatting\n(default_formats)"):::formatting
        end

        subgraph "Utilities"
            TC("Terminal Colors\n(bcolors)"):::utils
            NH("Number Helpers\n(number_helpers)"):::utils
        end
    end

    %% Testing Infrastructure
    subgraph "Testing Infrastructure"
        T("Tests Suite\n(Unit tests for all modules)"):::test
    end

    %% CI/CD & Documentation
    subgraph "CI/CD & Documentation"
        CICD("CI/CD Workflows & Config"):::cicd
    end

    %% Relationships within Chispa Core Library
    API --> CC
    API --> DC
    API --> RC
    API --> RSC
    API --> SC
    API --> SFC
    API --> FC
    API --> TC
    API --> NH

    FC --> DC
    TC --> FC
    NH --> DC
    EX1 --> DC

    %% Testing dependencies
    T --> API
    T --> CC
    T --> DC
    T --> RC
    T --> RSC
    T --> SC
    T --> SFC
    T --> FC
    T --> DF

    %% CI/CD triggering tests
    CICD --> T

    %% Click Events
    click API "https://github.com/mrpowers/chispa/blob/main/chispa/__init__.py"
    click CC "https://github.com/mrpowers/chispa/blob/main/chispa/column_comparer.py"
    click DC "https://github.com/mrpowers/chispa/blob/main/chispa/dataframe_comparer.py"
    click RC "https://github.com/mrpowers/chispa/blob/main/chispa/row_comparer.py"
    click RSC "https://github.com/mrpowers/chispa/blob/main/chispa/rows_comparer.py"
    click SC "https://github.com/mrpowers/chispa/blob/main/chispa/schema_comparer.py"
    click SFC "https://github.com/mrpowers/chispa/blob/main/chispa/structfield_comparer.py"
    click FC "https://github.com/mrpowers/chispa/blob/main/chispa/formatting/formatting_config.py"
    click FSH "https://github.com/mrpowers/chispa/blob/main/chispa/formatting/format_string.py"
    click FD "https://github.com/mrpowers/chispa/blob/main/chispa/formatting/formats.py"
    click DF "https://github.com/mrpowers/chispa/blob/main/chispa/default_formats.py"
    click TC "https://github.com/mrpowers/chispa/blob/main/chispa/bcolors.py"
    click NH "https://github.com/mrpowers/chispa/blob/main/chispa/number_helpers.py"
    click T "https://github.com/mrpowers/chispa/tree/main/tests"
    click CICD "https://github.com/mrpowers/chispa/blob/main/ci/environment-py39.yml"

    %% Styles
    classDef core fill:#FDEBD0,stroke:#D35400,stroke-width:2px,color:black
    classDef comparer fill:#FAD7A0,stroke:#E67E22,stroke-width:2px,color:black
    classDef formatting fill:#D1F2EB,stroke:#1ABC9C,stroke-width:2px,color:black
    classDef utils fill:#D6EAF8,stroke:#3498DB,stroke-width:2px,color:black
    classDef test fill:#F9E79F,stroke:#F1C40F,stroke-width:2px,color:black
    classDef cicd fill:#E8DAEF,stroke:#9B59B6,stroke-width:2px,color:black
    classDef external fill:#D5F5E3,stroke:#27AE60,stroke-width:2px,color:black
```
