# onsdigital/rdsa-utils architecture

> Architecture diagram of the GitHub repository onsdigital/rdsa-utils, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/onsdigital/rdsa-utils
- Repository: https://github.com/onsdigital/rdsa-utils

## Overview

##### Project type and purpose

- Project type: Open-source Python utility library (not a running service).
- Purpose: Provide reusable PySpark, Pandas, and general pipeline utilities for RDSA teams.
- Key platforms: Cloudera Data Platform (CDP), Google Cloud Platform (GCP), plus general Python/PySpark tasks.

##### File structure

Top-level folders & files:
- `rdsa_utils/` – core Python package
– `helpers/`, `io/`, `methods/`, `validation.py`, `logging.py`, `exceptions.py`
- `rdsa_utils/cdp/` – CDP-specific subpackage
– `helpers/` (HDFS, Impala, S3), `io/` (input, output, runlog)
- `rdsa_utils/gcp/` – GCP-specific subpackage
– `helpers/` (GCS, BigQuery), `io/` (inputs, outputs)
- Config & packaging: `setup.py`, `pyproject.toml`, `setup.cfg`
- Docs & CI/CD: `mkdocs.yml`, `.github/workflows/…`
- Tests: `tests/` mirrored under cdp, gcp, helpers, io, methods

##### README for additional insights

- Confirms separation of general utilities versus platform-specific modules.
- Highlights automated docs via MkDocs & CI pipelines publishing to PyPI and GitHub Pages.

##### Main components & relationships

a. Core utilities module (`rdsa_utils`):
- General helpers (Python, PySpark, log parsing)
- General IO (config, input, output)
- Common methods & validation & logging & exceptions

b. Platform-specific subpackages:
- CDP helpers & IO
- GCP helpers & IO
(Both built on top of core utilities)

c. External dependencies & services:
- PySpark, Pandas (data processing)
- HDFS, Impala, AWS S3 (CDP)
- GCS, BigQuery (GCP)

d. Supporting infrastructure:
- GitHub (version control)
- GitHub Actions (CI: test, lint; CD: publish to PyPI & docs)
- MkDocs (documentation generator)
- PyPI (distribution)

##### Architectural patterns & design principles

- Modular layering: core → platform adapters → consumers
- Single-responsibility modules (helpers vs IO vs methods)
- Extensibility via subpackages for each cloud platform
- Testability: each module accompanied by unit tests

##### Detailed component breakdown

- Code layers:
– Core utilities: `helpers`, `io`, `methods`, `validation`, `logging`, `exceptions`
– CDP: `cdp.helpers`, `cdp.io`
– GCP: `gcp.helpers`, `gcp.io`
- External libraries: PySpark, Pandas
- Cloud endpoints: HDFS, Impala, S3, GCS, BigQuery
- Packaging & publication: setup.py, PyPI
- Documentation: MkDocs → GitHub Pages
- CI/CD: GitHub Actions workflows

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Utilities
    subgraph "Core Utilities" 
        direction TB
        Helpers["helpers/"]:::code
        IO_Config["io/config.py"]:::code
        IO_Input["io/input.py"]:::code
        IO_Output["io/output.py"]:::code
        Methods["methods/averaging_methods.py"]:::code
        Validation["validation.py"]:::code
        Logging["logging.py"]:::code
        Exceptions["exceptions.py"]:::code
    end

    %% CDP Subpackage
    subgraph "CDP Subpackage" 
        direction TB
        CDP_HDFS["helpers/hdfs_utils.py"]:::code
        CDP_Impala["helpers/impala.py"]:::code
        CDP_S3["helpers/s3_utils.py"]:::code
        CDP_IO_Input["io/input.py"]:::code
        CDP_IO_Output["io/output.py"]:::code
        CDP_RunLog["io/pipeline_runlog.py"]:::code
    end

    %% GCP Subpackage
    subgraph "GCP Subpackage"
        direction TB
        GCP_Utils["helpers/gcp_utils.py"]:::code
        GCP_IO_Inputs["io/inputs.py"]:::code
        GCP_IO_Outputs["io/outputs.py"]:::code
    end

    %% PySpark Integration
    PySpark_Util["pyspark.py"]:::code
    PySpark_Parser["pyspark_log_parser/"]:::code

    %% External Libraries
    subgraph "External Libraries"
        direction TB
        PySpark["PySpark"]:::external
        Pandas["Pandas"]:::external
    end

    %% Cloud Services
    subgraph "Cloud Services"
        direction TB
        HDFS["HDFS"]:::service
        Impala["Impala"]:::service
        S3["S3"]:::service
        GCS["GCS"]:::service
        BigQuery["BigQuery"]:::service
    end

    %% CI/CD & Docs
    subgraph "CI/CD & Docs" 
        direction TB
        GitHub["GitHub"]:::ci
        Actions["GitHub Actions"]:::ci
        MkDocs["MkDocs"]:::ci
        PyPI["PyPI"]:::registry
        GH_Pages["GitHub Pages"]:::registry
    end

    %% Dependencies
    CDP_HDFS -->|depends on| Helpers
    CDP_Impala -->|depends on| Helpers
    CDP_S3 -->|depends on| Helpers
    CDP_IO_Input -->|depends on| IO_Input
    CDP_IO_Output -->|depends on| IO_Output
    CDP_RunLog -->|depends on| IO_Config

    GCP_Utils -->|depends on| Helpers
    GCP_IO_Inputs -->|depends on| IO_Input
    GCP_IO_Outputs -->|depends on| IO_Output

    Helpers -->|uses| PySpark_Util
    Helpers -->|uses| PySpark_Parser
    IO_Config -->|uses| Pandas
    Methods -->|uses| Pandas
    Validation -->|validates| IO_Config
    Logging -->|logs to| IO_Output
    Exceptions -->|raised by| Methods

    CDP_HDFS -->|reads from| HDFS
    CDP_Impala -->|queries| Impala
    CDP_S3 -->|reads/writes| S3
    GCP_Utils -->|interacts with| GCS
    GCP_Utils -->|writes to| BigQuery
    GCP_IO_Outputs -->|writes to| BigQuery

    %% CI/CD Flow
    GitHub -->|on commit| Actions
    Actions -->|run tests & lint| Tests["Tests"]:::ci
    Actions -->|build & package| Package["Package"]:::ci
    Package -->|publish| PyPI
    MkDocs -->|generate| Docs["Docs Site"]:::ci
    Docs -->|deploy| GH_Pages

    %% Click Events - Core Utilities
    click Helpers "https://github.com/onsdigital/rdsa-utils/tree/main/rdsa_utils/helpers/"
    click IO_Config "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/io/config.py"
    click IO_Input "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/io/input.py"
    click IO_Output "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/io/output.py"
    click Methods "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/methods/averaging_methods.py"
    click Validation "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/validation.py"
    click Logging "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/logging.py"
    click Exceptions "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/exceptions.py"

    %% Click Events - CDP
    click CDP_HDFS "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/cdp/helpers/hdfs_utils.py"
    click CDP_Impala "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/cdp/helpers/impala.py"
    click CDP_S3 "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/cdp/helpers/s3_utils.py"
    click CDP_IO_Input "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/cdp/io/input.py"
    click CDP_IO_Output "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/cdp/io/output.py"
    click CDP_RunLog "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/cdp/io/pipeline_runlog.py"

    %% Click Events - GCP
    click GCP_Utils "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/gcp/helpers/gcp_utils.py"
    click GCP_IO_Inputs "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/gcp/io/inputs.py"
    click GCP_IO_Outputs "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/gcp/io/outputs.py"

    %% Click Events - PySpark
    click PySpark_Util "https://github.com/onsdigital/rdsa-utils/blob/main/rdsa_utils/helpers/pyspark.py"
    click PySpark_Parser "https://github.com/onsdigital/rdsa-utils/tree/main/rdsa_utils/helpers/pyspark_log_parser/"

    %% Click Events - Packaging & Docs & CI
    click PyPI "https://github.com/onsdigital/rdsa-utils/blob/main/setup.py"
    click PyPI "https://github.com/onsdigital/rdsa-utils/blob/main/pyproject.toml"
    click PyPI "https://github.com/onsdigital/rdsa-utils/blob/main/setup.cfg"
    click MkDocs "https://github.com/onsdigital/rdsa-utils/blob/main/mkdocs.yml"
    click Docs "https://github.com/onsdigital/rdsa-utils/tree/main/docs/"
    click Actions "https://github.com/onsdigital/rdsa-utils/blob/main/.github/workflows/pull_request_workflow.yaml"
    click Actions "https://github.com/onsdigital/rdsa-utils/blob/main/.github/workflows/deploy_mkdocs.yaml"
    click Actions "https://github.com/onsdigital/rdsa-utils/blob/main/.github/workflows/deploy_pypi.yaml"

    %% Styles
    classDef code fill:#D6E8FF,stroke:#0366D6,color:#034C8C
    classDef external fill:#E8F5E9,stroke:#388E3C,color:#256029
    classDef service fill:#E3F2FD,stroke:#1976D2,color:#0D47A1
    classDef ci fill:#F5F5F5,stroke:#9E9E9E,color:#616161
    classDef registry fill:#FFF3E0,stroke:#FB8C00,color:#E65100
```
