# iamraymondlow/poi-conflation-framework architecture

> Architecture diagram of the GitHub repository iamraymondlow/poi-conflation-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iamraymondlow/poi-conflation-framework
- Repository: https://github.com/iamraymondlow/poi-conflation-framework

## Overview

##### Project Type and Purpose
- The project is an end-to-end Point of Interest (POI) conflation framework written in Python.
- Its primary goal is to merge (or conflate) POI data from multiple sources (e.g., Google Maps, HERE Maps, OneMap, OpenStreetMap, SLA) by performing several steps: data procurement, schema standardisation, taxonomy mapping, matching, unification, and data verification.
- The README confirms the six-step process and indicates that the framework has been applied in a geospatial context, particularly in urban planning.

##### Analyzing the File Structure
- At the top level, we see several Python modules that are clearly dedicated to extracting POIs from different sources:
– googlemap_poi_extractor.py
– heremap_poi_extractor.py
– onemap_poi_extractor.py
– osm_poi_extractor.py
– sla_poi_extractor.py
- Additional modules include:
– poi_matching_unification.py (this likely handles the matching and unification stage)
– taxonomy_mapper.py (which probably deals with schema or taxonomy mapping between different data sources)
– helper_functions.py (supporting utility functions used by the main modules)
- There are supporting directories that contain boundary data (“master-plan-2014-planning-area-boundary-web” and “master-plan-2014-subzone-boundary-no-sea”), which provide spatial context for the POI data.
- The presence of zipped files (in KML and shapefile formats) and metadata indicates that geospatial boundary data is a significant input to the process.

##### Key Components
a. Data Procurement / Extraction:
– List separate modules for each POI extractor (Google, HERE, OneMap, OSM, SLA).
– Indicate that they ingest raw POI data from different external sources.
b. Schema Standardisation:
– Although there might not be a dedicated module file for this, you may group preprocessing functionality from helper_functions.py or integrate it as part of the pipeline for preparing data.
c. Taxonomy Mapping:
– Represent the taxonomy_mapper.py as a key component that aligns different POI attribute schemas.
d. POI Matching:
e. POI Unification:
– Either part of the matching module or represented as a separate step, show the function that unifies the identified duplicate or matching POI entries into a single, consolidated record.
f. Data Verification:
– Although not a specific file, include the final validation stage of the pipeline to check data quality.
g. Supporting Geospatial Boundary Data:

##### Architectural Relationships and Interactions
- Each extractor outputs raw POI data (these outputs may be standardized or normalized immediately).
- Feed all raw data into a “Schema Standardisation” layer (which may be a collection of helper functions).
- The standardized data is then routed to the “Taxonomy Mapping” component to harmonize the underlying classification.
- Once taxonomy mapping is complete, all datasets are combined and forwarded into the “POI Matching & Unification” module, which effectively identifies duplicates and creates a unified POI dataset.
- Finally, include a verification/check component (which can be included as part of the matching module or represented separately) to ensure data quality.
- You may also illustrate how the geospatial boundary files are used (for example, as a spatial filter, or to tag POIs with planning area metadata).

##### Architectural Patterns and Design Principles
- Modularity: Each major step (extraction, standardisation, mapping, matching) is in its own module.
- Pipeline/Data Flow Architecture: Data flows sequentially through each processing stage.
- Separation of Concerns: Different POI data sources are handled by dedicated extractor modules.
- Extensibility: The design allows for additional data sources to be integrated into the pipeline with minimal disruption.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "POI Data Sources"
        GMap["Google Maps POI Extractor"]:::extractor
        Heremap["HERE Maps POI Extractor"]:::extractor
        OneMap["OneMap POI Extractor"]:::extractor
        OSM["OpenStreetMap POI Extractor"]:::extractor
        SLA["SLA POI Extractor"]:::extractor
    end

    subgraph "Geospatial Boundaries"
        BoundaryPlanning["Master Plan Boundaries (Planning Area)"]:::boundary
        BoundarySubzone["Master Plan Boundaries (Subzone)"]:::boundary
    end

    subgraph "Data Transformation Pipeline"
        SchemaStd["Schema Standardisation Layer"]:::transform
        TaxMap["Taxonomy Mapping Component"]:::transform
    end

    Matching["POI Matching & Unification Module"]:::matching
    Verification["Data Verification / Quality Assurance"]:::verification

    %% Connections from Extractors to Schema Standardisation
    GMap -->|"rawPOIdata"| SchemaStd
    Heremap -->|"rawPOIdata"| SchemaStd
    OneMap -->|"rawPOIdata"| SchemaStd
    OSM -->|"rawPOIdata"| SchemaStd
    SLA -->|"rawPOIdata"| SchemaStd

    %% Transformation Pipeline Flow
    SchemaStd -->|"standardisedData"| TaxMap
    TaxMap -->|"mappedData"| Matching

    %% Boundaries feed into Matching Module
    BoundaryPlanning -->|"spatialData"| Matching
    BoundarySubzone -->|"spatialData"| Matching

    %% Matching to Data Verification
    Matching -->|"matchedRecords"| Verification

    %% Click Events
    click GMap "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/googlemap_poi_extractor.py"
    click Heremap "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/heremap_poi_extractor.py"
    click OneMap "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/onemap_poi_extractor.py"
    click OSM "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/osm_poi_extractor.py"
    click SLA "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/sla_poi_extractor.py"
    click SchemaStd "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/helper_functions.py"
    click TaxMap "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/taxonomy_mapper.py"
    click Matching "https://github.com/iamraymondlow/poi-conflation-framework/blob/main/poi_matching_unification.py"
    click BoundaryPlanning "https://github.com/iamraymondlow/poi-conflation-framework/tree/main/master-plan-2014-planning-area-boundary-web/"
    click BoundarySubzone "https://github.com/iamraymondlow/poi-conflation-framework/tree/main/master-plan-2014-subzone-boundary-no-sea/"

    %% Styles
    classDef extractor fill:#f9c,stroke:#333,stroke-width:2px;
    classDef transform fill:#cff,stroke:#333,stroke-width:2px;
    classDef matching fill:#cfc,stroke:#333,stroke-width:2px;
    classDef verification fill:#fcc,stroke:#333,stroke-width:2px;
    classDef boundary fill:#fc9,stroke:#333,stroke-width:2px;
```
