# onsdigital/research-and-development architecture

> Architecture diagram of the GitHub repository onsdigital/research-and-development, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/onsdigital/research-and-development
- Repository: https://github.com/onsdigital/research-and-development

## Overview

##### Project Type and Purpose
- Type: A modular ETL/batch data‐processing pipeline written in Python.
- Purpose: Ingest raw survey/accounting data, progressively clean, map, estimate and impute missing values, detect outliers, apply territorial apportionment, freeze results, and finally produce national/regional output files for national accounts.

##### File Structure
- Top‐level “src” directory broken into subpackages:
– staging: raw data loading & basic validation
– construction: combining and validating multiple inputs
– mapping: code lookups (ITL, SIC, postcode, etc.)
– estimation: weight calculations and national/regional estimates
– imputation: filling gaps via different methods (MoR, TMI, manual)
– freezing: lock‐down snapshots and compare changes
– outlier_detection: automated & manual QA filters
– site_apportionment: allocate outputs to sites or geographies
– outputs: final file‐exporters & formats (CSV, SAS, etc.)
- Cross‐cutting: utils, helpers, config and TOML schemas, main orchestration scripts (pipeline.py, main.py, export_main.py).
- Data folders: data/raw → data/interim → data/processed; outputs/ for final artifacts.
- CI/CD: Jenkinsfile, GitHub Actions workflow, flake8, pre‐commit, pytest.

##### README
- Confirms batch processing for national/regional R&D spend
- Uses govcookiecutter template for data projects
- MIT‐licensed, Sphinx docs under docs/ (structure, reference, user guide)

##### Main Components and Their Interactions
A. Data Sources (Inputs)
– Local FS or HDFS/S3 (via utils/hdfs_mods.py, utils/s3_mods.py)
– data/raw directory
B. Configuration
– TOML/YAML schemas under config/ and src/dev_config.yaml
– Central config loader in src/utils/config.py
C. Pipeline Orchestrator
– pipeline.py (master driver)
– main.py and export_main.py entry points
D. Staging Layer
– staging_main.py: basic validations, snapshot processing
1. Construction: src/construction/* → aggregate and validate
2. Mapping: src/mapping/* → attach code‐based lookups
3. Estimation: src/estimation/* → calculate weights, apply them
4. Imputation: src/imputation/* → various impute methods
5. Freezing: src/freezing/* → apply snapshot logic, compare
6. Outlier Detection: src/outlier_detection/* → QA filters
7. Site Apportionment: src/site_apportionment/* → geog allocation
F. Output Generation
– src/outputs/* → export files in required formats
– outputs/ directory, plus optional S3/HDFS via utils
G. Logging & Monitoring
– src/utils/logger.py, logs/run_logs

Raw Data → Staging → Construction → Mapping → Estimation → Imputation → Freezing → Outlier Detection → Site Apportionment → Outputs

##### Architectural Patterns
- Modular “stage” pattern: each processing area isolated in its own package with a main entrypoint (`*_main.py`).
- Config‐driven: TOML/YAML schemas validate inputs and output formats.
- Shared utilities: common helpers and wrappers in src/utils and helpers/.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Sources"
        HDFS["HDFS/S3 Loader\n(utils/hdfs_mods.py)"]:::processing
        S3["S3 Loader\n(utils/s3_mods.py)"]:::processing
        RawData["Raw Data\n(data/raw/)"]:::storage
    end

    subgraph "Configuration"
        ConfigLoader["Config Loader\n(src/utils/config.py)"]:::config
        DevConfig["Dev Config\n(src/dev_config.yaml)"]:::config
        Schemas["Schemas\n(config/)"]:::config
    end

    subgraph "Orchestrator"
        Pipeline["Pipeline Driver\n(src/pipeline.py)"]:::orchestrator
        MainEntry["Main Entry\n(main.py)"]:::orchestrator
        ExportEntry["Export Entry\n(export_main.py)"]:::orchestrator
    end

    subgraph "Pipeline Stages"
        Staging["Staging Module\n(src/staging/staging_main.py)"]:::processing
        Construction["Construction Module\n(src/construction/construction_main.py)"]:::processing
        Mapping["Mapping Module\n(src/mapping/mapping_main.py)"]:::processing
        Estimation["Estimation Module\n(src/estimation/estimation_main.py)"]:::processing
        Imputation["Imputation Module\n(src/imputation/imputation_main.py)"]:::processing
        Freezing["Freezing Module\n(src/freezing/freezing_main.py)"]:::processing
        Outlier["Outlier Detection\n(src/outlier_detection/outlier_main.py)"]:::processing
        Apportionment["Site Apportionment\n(src/site_apportionment/site_apportionment_main.py)"]:::processing
    end

    subgraph "Output Generation"
        OutputsMain["Outputs Module\n(src/outputs/outputs_main.py)"]:::processing
        FinalOutputs["Final Outputs\n(outputs/)"]:::storage
    end

    subgraph "Logging & Monitoring"
        Logger["Logger\n(src/utils/logger.py)"]:::logging
        RunLogs["Run Logs\n(logs/run_logs/)"]:::logging
    end

    HDFS --> RawData
    S3 --> RawData
    RawData --> Staging
    Staging --> Construction
    Construction --> Mapping
    Mapping --> Estimation
    Estimation --> Imputation
    Imputation --> Freezing
    Freezing --> Outlier
    Outlier --> Apportionment
    Apportionment --> OutputsMain
    OutputsMain --> FinalOutputs

    MainEntry --> Pipeline
    Pipeline --> Staging
    Pipeline --> Construction
    Pipeline --> Mapping
    Pipeline --> Estimation
    Pipeline --> Imputation
    Pipeline --> Freezing
    Pipeline --> Outlier
    Pipeline --> Apportionment
    Pipeline --> OutputsMain
    ExportEntry --> OutputsMain

    ConfigLoader -.-> Staging
    ConfigLoader -.-> Construction
    ConfigLoader -.-> Mapping
    ConfigLoader -.-> Estimation
    ConfigLoader -.-> Imputation
    ConfigLoader -.-> Freezing
    ConfigLoader -.-> Outlier
    ConfigLoader -.-> Apportionment
    ConfigLoader -.-> OutputsMain

    Logger -.-> Staging
    Logger -.-> Construction
    Logger -.-> Mapping
    Logger -.-> Estimation
    Logger -.-> Imputation
    Logger -.-> Freezing
    Logger -.-> Outlier
    Logger -.-> Apportionment
    Logger -.-> OutputsMain

    classDef processing fill:#cfe2f3,stroke:#036;stroke-width:1px
    classDef storage fill:#f9d5e5,stroke:#a00;stroke-width:1px,stroke-dasharray: 5 5
    classDef config fill:#d9ead3,stroke:#080;stroke-width:1px
    classDef orchestrator fill:#fff2cc,stroke:#aa0;stroke-width:1px
    classDef logging fill:#f4cccc,stroke:#a00;stroke-width:1px

    click HDFS "https://github.com/onsdigital/research-and-development/blob/develop/src/utils/hdfs_mods.py"
    click S3 "https://github.com/onsdigital/research-and-development/blob/develop/src/utils/s3_mods.py"
    click RawData "https://github.com/onsdigital/research-and-development/tree/develop/data/raw/"
    click ConfigLoader "https://github.com/onsdigital/research-and-development/blob/develop/src/utils/config.py"
    click DevConfig "https://github.com/onsdigital/research-and-development/blob/develop/src/dev_config.yaml"
    click Schemas "https://github.com/onsdigital/research-and-development/tree/develop/config/"
    click Pipeline "https://github.com/onsdigital/research-and-development/blob/develop/src/pipeline.py"
    click MainEntry "https://github.com/onsdigital/research-and-development/blob/develop/main.py"
    click ExportEntry "https://github.com/onsdigital/research-and-development/blob/develop/export_main.py"
    click Staging "https://github.com/onsdigital/research-and-development/blob/develop/src/staging/staging_main.py"
    click Construction "https://github.com/onsdigital/research-and-development/blob/develop/src/construction/construction_main.py"
    click Mapping "https://github.com/onsdigital/research-and-development/blob/develop/src/mapping/mapping_main.py"
    click Estimation "https://github.com/onsdigital/research-and-development/blob/develop/src/estimation/estimation_main.py"
    click Imputation "https://github.com/onsdigital/research-and-development/blob/develop/src/imputation/imputation_main.py"
    click Freezing "https://github.com/onsdigital/research-and-development/blob/develop/src/freezing/freezing_main.py"
    click Outlier "https://github.com/onsdigital/research-and-development/blob/develop/src/outlier_detection/outlier_main.py"
    click Apportionment "https://github.com/onsdigital/research-and-development/blob/develop/src/site_apportionment/site_apportionment_main.py"
    click OutputsMain "https://github.com/onsdigital/research-and-development/blob/develop/src/outputs/outputs_main.py"
    click Logger "https://github.com/onsdigital/research-and-development/blob/develop/src/utils/logger.py"
    click RunLogs "https://github.com/onsdigital/research-and-development/tree/develop/logs/run_logs/"
```
