How onsdigital/rdsa-utils works
Project type and purpose
• Project type: Open-source Python utility library (not a running service).
• Purpose: Provide reusable PySpark, Pandas, and general pipeline utilities for RDSA teams.
• Key platforms: Cloudera Data Platform (CDP), Google Cloud Platform (GCP), plus general Python/PySpark tasks.
File structure
Top-level folders & files:
• rdsa_utils/ – core Python package
– helpers/, io/, methods/, validation.py, logging.py, exceptions.py
• rdsa_utils/cdp/ – CDP-specific subpackage
– helpers/ (HDFS, Impala, S3), io/ (input, output, runlog)
• rdsa_utils/gcp/ – GCP-specific subpackage
– helpers/ (GCS, BigQuery), io/ (inputs, outputs)
• Config & packaging: setup.py, pyproject.toml, setup.cfg
• Docs & CI/CD: mkdocs.yml, .github/workflows/…
• Tests: tests/ mirrored under cdp, gcp, helpers, io, methods
README for additional insights
• Confirms separation of general utilities versus platform-specific modules.
• Highlights automated docs via MkDocs & CI pipelines publishing to PyPI and GitHub Pages.
Main components & relationships
a. Core utilities module (rdsa_utils):
• General helpers (Python, PySpark, log parsing)
• General IO (config, input, output)
• Common methods & validation & logging & exceptions
b. Platform-specific subpackages:
• CDP helpers & IO
• GCP helpers & IO
(Both built on top of core utilities)
c. External dependencies & services:
• PySpark, Pandas (data processing)
• HDFS, Impala, AWS S3 (CDP)
• GCS, BigQuery (GCP)
d. Supporting infrastructure:
• GitHub (version control)
• GitHub Actions (CI: test, lint; CD: publish to PyPI & docs)
• MkDocs (documentation generator)
• PyPI (distribution)
Architectural patterns & design principles
• Modular layering: core → platform adapters → consumers
• Single-responsibility modules (helpers vs IO vs methods)
• Extensibility via subpackages for each cloud platform
• Testability: each module accompanied by unit tests
Detailed component breakdown
• Code layers:
– Core utilities: helpers, io, methods, validation, logging, exceptions
– CDP: cdp.helpers, cdp.io
– GCP: gcp.helpers, gcp.io
• External libraries: PySpark, Pandas
• Cloud endpoints: HDFS, Impala, S3, GCS, BigQuery
• Packaging & publication: setup.py, PyPI
• Documentation: MkDocs → GitHub Pages
• CI/CD: GitHub Actions workflows