# onsdigital/aims-spark architecture

> Architecture diagram of the GitHub repository onsdigital/aims-spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/onsdigital/aims-spark
- Repository: https://github.com/onsdigital/aims-spark

## Overview

##### Project Type and Purpose
- This is a back-end batch data pipeline (an ETL job) written in Scala/Spark. Its sole purpose is to read AddressBase Premium change-only update (COU) CSV files (and a hierarchy SQL table), transform and enrich them, then write the results as four different Elasticsearch indices (full/historic, full/non-historic, skinny, skinny/historic).

##### High-Level Architecture Components
a. Input Data Sources
- Flat files (CSV): BLPU, classification, cross-reference, delivery-point, hierarchy, LPI, organisation, street, street-descriptor, RDMF, plus optional island files.
- SQL Hierarchy table (populated via dbscripts/hierarchy_script.sql).
- (Optional) IDS lookup file mapping UPRN to AddressEntryId.
b. Configuration
- reference.conf (defaults) + application.conf (overrides) govern file locations, Spark settings, ES endpoints, feature flags (e.g. skinny, hybridNoHist, islands, custCodes).
c. Processing Engine
- Apache Spark job (Main.scala) launched on local mode, YARN, or Google Dataproc (Serverless v2.2+).
- Modules:
– SparkProvider: bootstraps SparkSession, Hadoop/GCS connector configs.
– Readers (AddressIndexFileReader): loads CSVs into DataFrames/RDDs.
– Models (CSVSchemas, EsDocument, HybridAddress, etc.) define schemas and case classes.
– Utils (Mappings, CensusClassificationHelper, SqlHelper, StringUtil, AuthUtil): transformation logic, lookups, classification, UPRN ↔ AddressEntryId mapping.
– Writers (ElasticSearchWriter): pushes DataFrames to Elasticsearch via elasticsearch-spark connector.
d. Output Data Store
- Elasticsearch cluster (version 8.14.x): four indices, controlled at runtime by flags (--skinny, --hybridNoHist, --custCodes).
e. Build & Deployment
- SBT build (build.sbt + sbt-assembly for fat-jar).
- CI/CD via Travis CI, Codecov, Codacy.
- Execution: `java -Dconfig.file=application.conf -jar ons-ai-batch-assembly.jar`.

##### Component Relationships and Data Flows
1. Job launcher (CLI or Dataproc API) passes runtime flags and config file.
2. Main.scala reads reference.conf/application.conf → constructs SparkSession (SparkProvider).
3. Spark reads each CSV (and dummy or real hierarchy SQL table via Spark JDBC or pre-exported CSV).
4. DataFrames flow through transformation pipeline:
- Schema enforcement (CSVSchemas) → enrichment/classification (CensusClassificationHelper, Mappings) → hybrid address generation → filtering by custCodes/historic/skinny logic.
5. ElasticSearchWriter writes final RDD/DataFrame partitions to ES nodes (via elasticsearch-spark).
6. CI pipeline runs unit tests under batch/src/test (ScalaTest), using dummy test CSVs and text files.

##### Architectural Patterns & Principles
- Modular ETL stages: readers → transformations → writers.
- Configuration-driven behavior: environment-agnostic.
- Immutable data processing in Spark RDD/DataFrame API.
- Separation of concerns: each utility class has a single responsibility (e.g. SQL helper, string util).
- Testability: unit tests for models, readers, utils.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "INPUT DATA SOURCES"
        direction TB
        CSVFiles["Flat Files (CSV)"]:::datastore
        SQLTable["Hierarchy SQL Table"]:::datastore
        IDSLookup["Optional IDS Lookup File"]:::datastore
    end

    HierarchyScript["dbscripts/hierarchy_script.sql"]:::datastore

    %% Configuration
    Config["Configuration (reference.conf, application.conf, log4j.properties)"]:::config

    %% Execution Targets
    subgraph "Execution Environments"
        direction TB
        Local["Local Mode"]:::compute
        Yarn["YARN"]:::compute
        Dataproc["Google Dataproc"]:::compute
    end

    %% Spark Cluster
    subgraph "Spark Cluster" 
        direction TB
        MainApp["Main.scala (Job Launcher)"]:::compute
        SparkProv["SparkProvider.scala"]:::compute

        subgraph "ETL Modules"
            direction TB
            Readers["AddressIndexFileReader"]:::compute
            subgraph "Models"
                direction TB
                CSVSchemas["CSVSchemas"]:::compute
                EsDocument["EsDocument"]:::compute
                HybridAddress["HybridAddress"]:::compute
                HybridAddressEsDoc["HybridAddressEsDocument"]:::compute
                HybridAddressSkinnyEsDoc["HybridAddressSkinnyEsDocument"]:::compute
                HybridAddressSkinny["HybridAddressSkinny"]:::compute
            end
            subgraph "Utils"
                direction TB
                CensusClassHelper["CensusClassificationHelper"]:::compute
                Mappings["Mappings"]:::compute
                SqlHelper["SqlHelper"]:::compute
                StringUtil["StringUtil"]:::compute
                AuthUtil["AuthUtil"]:::compute
            end
            Writers["ElasticSearchWriter"]:::compute
        end
    end

    %% Output
    ESCluster["Elasticsearch Cluster"]:::datastore

    %% Data Flows
    HierarchyScript -->|populates| SQLTable
    CSVFiles -->|read CSVs into DataFrames| Readers
    SQLTable -->|read via JDBC/CSV| Readers
    IDSLookup -->|optional lookup| Readers

    Config -->|injection| MainApp
    MainApp -->|initialises| SparkProv
    SparkProv --> Readers

    Readers -->|schema enforcement| CSVSchemas
    CSVSchemas -->|enrichment| CensusClassHelper
    CensusClassHelper --> Mappings
    Mappings --> HybridAddress
    HybridAddress --> HybridAddressEsDoc
    HybridAddress --> HybridAddressSkinny
    HybridAddressEsDoc --> Writers
    HybridAddressSkinny --> HybridAddressSkinnyEsDoc
    HybridAddressSkinnyEsDoc --> Writers

    Writers -->|bulk write to ES| ESCluster

    MainApp --> Local
    MainApp --> Yarn
    MainApp --> Dataproc

    %% Click Events
    click Readers "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/readers/AddressIndexFileReader.scala"
    click HierarchyScript "https://github.com/onsdigital/aims-spark/blob/main/dbscripts/hierarchy_script.sql"
    click Config "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/resources/reference.conf"
    click MainApp "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/Main.scala"
    click SparkProv "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/utils/SparkProvider.scala"
    click CSVSchemas "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/models/CSVSchemas.scala"
    click EsDocument "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/models/EsDocument.scala"
    click HybridAddress "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/models/HybridAddress.scala"
    click HybridAddressEsDoc "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/models/HybridAddressEsDocument.scala"
    click HybridAddressSkinnyEsDoc "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/models/HybridAddressSkinnyEsDocument.scala"
    click HybridAddressSkinny "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/models/HybridAddressSkinny.scala"
    click CensusClassHelper "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/utils/CensusClassificationHelper.scala"
    click Mappings "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/utils/Mappings.scala"
    click SqlHelper "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/utils/SqlHelper.scala"
    click StringUtil "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/utils/StringUtil.scala"
    click AuthUtil "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/utils/AuthUtil.scala"
    click Writers "https://github.com/onsdigital/aims-spark/blob/main/batch/src/main/scala/uk/gov/ons/addressindex/writers/ElasticSearchWriter.scala"

    %% Styles
    classDef datastore fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef compute fill:#ffcc80,stroke:#fb8c00,color:#e65100
    classDef config fill:#e0e0e0,stroke:#9e9e9e,color:#424242
```
