How onsdigital/aims-spark works
Project Type and Purpose
• This is a back-end batch data pipeline (an ETL job) written in Scala/Spark. Its sole purpose is to read AddressBase Premium change-only update (COU) CSV files (and a hierarchy SQL table), transform and enrich them, then write the results as four different Elasticsearch indices (full/historic, full/non-historic, skinny, skinny/historic).
High-Level Architecture Components
a. Input Data Sources
• Flat files (CSV): BLPU, classification, cross-reference, delivery-point, hierarchy, LPI, organisation, street, street-descriptor, RDMF, plus optional island files.
• SQL Hierarchy table (populated via dbscripts/hierarchy_script.sql).
• (Optional) IDS lookup file mapping UPRN to AddressEntryId.
b. Configuration
• reference.conf (defaults) + application.conf (overrides) govern file locations, Spark settings, ES endpoints, feature flags (e.g. skinny, hybridNoHist, islands, custCodes).
c. Processing Engine
• Apache Spark job (Main.scala) launched on local mode, YARN, or Google Dataproc (Serverless v2.2+).
• Modules:
– SparkProvider: bootstraps SparkSession, Hadoop/GCS connector configs.
– Readers (AddressIndexFileReader): loads CSVs into DataFrames/RDDs.
– Models (CSVSchemas, EsDocument, HybridAddress, etc.) define schemas and case classes.
– Utils (Mappings, CensusClassificationHelper, SqlHelper, StringUtil, AuthUtil): transformation logic, lookups, classification, UPRN ↔ AddressEntryId mapping.
– Writers (ElasticSearchWriter): pushes DataFrames to Elasticsearch via elasticsearch-spark connector.
d. Output Data Store
• Elasticsearch cluster (version 8.14.x): four indices, controlled at runtime by flags (--skinny, --hybridNoHist, --custCodes).
e. Build & Deployment
• SBT build (build.sbt + sbt-assembly for fat-jar).
• CI/CD via Travis CI, Codecov, Codacy.
• Execution: java -Dconfig.file=application.conf -jar ons-ai-batch-assembly.jar.
Component Relationships and Data Flows
1. Job launcher (CLI or Dataproc API) passes runtime flags and config file.
2. Main.scala reads reference.conf/application.conf → constructs SparkSession (SparkProvider).
3. Spark reads each CSV (and dummy or real hierarchy SQL table via Spark JDBC or pre-exported CSV).
4. DataFrames flow through transformation pipeline:
• Schema enforcement (CSVSchemas) → enrichment/classification (CensusClassificationHelper, Mappings) → hybrid address generation → filtering by custCodes/historic/skinny logic.
5. ElasticSearchWriter writes final RDD/DataFrame partitions to ES nodes (via elasticsearch-spark).
6. CI pipeline runs unit tests under batch/src/test (ScalaTest), using dummy test CSVs and text files.
Architectural Patterns & Principles
• Modular ETL stages: readers → transformations → writers.
• Configuration-driven behavior: environment-agnostic.
• Immutable data processing in Spark RDD/DataFrame API.
• Separation of concerns: each utility class has a single responsibility (e.g. SQL helper, string util).
• Testability: unit tests for models, readers, utils.