How onsdigital/research-and-development works
Project Type and Purpose
• Type: A modular ETL/batch data‐processing pipeline written in Python.
• Purpose: Ingest raw survey/accounting data, progressively clean, map, estimate and impute missing values, detect outliers, apply territorial apportionment, freeze results, and finally produce national/regional output files for national accounts.
File Structure
• Top‐level “src” directory broken into subpackages:
– staging: raw data loading & basic validation
– construction: combining and validating multiple inputs
– mapping: code lookups (ITL, SIC, postcode, etc.)
– estimation: weight calculations and national/regional estimates
– imputation: filling gaps via different methods (MoR, TMI, manual)
– freezing: lock‐down snapshots and compare changes
– outlier_detection: automated & manual QA filters
– site_apportionment: allocate outputs to sites or geographies
– outputs: final file‐exporters & formats (CSV, SAS, etc.)
• Cross‐cutting: utils, helpers, config and TOML schemas, main orchestration scripts (pipeline.py, main.py, export_main.py).
• Data folders: data/raw → data/interim → data/processed; outputs/ for final artifacts.
• CI/CD: Jenkinsfile, GitHub Actions workflow, flake8, pre‐commit, pytest.
README
• Confirms batch processing for national/regional R&D spend
• Uses govcookiecutter template for data projects
• MIT‐licensed, Sphinx docs under docs/ (structure, reference, user guide)
Main Components and Their Interactions
A. Data Sources (Inputs)
– Local FS or HDFS/S3 (via utils/hdfs_mods.py, utils/s3_mods.py)
– data/raw directory
B. Configuration
– TOML/YAML schemas under config/ and src/dev_config.yaml
– Central config loader in src/utils/config.py
C. Pipeline Orchestrator
– pipeline.py (master driver)
– main.py and export_main.py entry points
D. Staging Layer
– staging_main.py: basic validations, snapshot processing
1. Construction: src/construction/* → aggregate and validate
2. Mapping: src/mapping/* → attach code‐based lookups
3. Estimation: src/estimation/* → calculate weights, apply them
4. Imputation: src/imputation/* → various impute methods
5. Freezing: src/freezing/* → apply snapshot logic, compare
6. Outlier Detection: src/outlier_detection/* → QA filters
7. Site Apportionment: src/site_apportionment/* → geog allocation
F. Output Generation
– src/outputs/* → export files in required formats
– outputs/ directory, plus optional S3/HDFS via utils
G. Logging & Monitoring
– src/utils/logger.py, logs/run_logs
Raw Data → Staging → Construction → Mapping → Estimation → Imputation → Freezing → Outlier Detection → Site Apportionment → Outputs
Architectural Patterns
• Modular “stage” pattern: each processing area isolated in its own package with a main entrypoint (*_main.py).
• Config‐driven: TOML/YAML schemas validate inputs and output formats.
• Shared utilities: common helpers and wrappers in src/utils and helpers/.