How iamraymondlow/poi-conflation-framework works
Project Type and Purpose
• The project is an end-to-end Point of Interest (POI) conflation framework written in Python.
• Its primary goal is to merge (or conflate) POI data from multiple sources (e.g., Google Maps, HERE Maps, OneMap, OpenStreetMap, SLA) by performing several steps: data procurement, schema standardisation, taxonomy mapping, matching, unification, and data verification.
• The README confirms the six-step process and indicates that the framework has been applied in a geospatial context, particularly in urban planning.
Analyzing the File Structure
• At the top level, we see several Python modules that are clearly dedicated to extracting POIs from different sources:
– googlemap_poi_extractor.py
– heremap_poi_extractor.py
– onemap_poi_extractor.py
– osm_poi_extractor.py
– sla_poi_extractor.py
• Additional modules include:
– poi_matching_unification.py (this likely handles the matching and unification stage)
– taxonomy_mapper.py (which probably deals with schema or taxonomy mapping between different data sources)
– helper_functions.py (supporting utility functions used by the main modules)
• There are supporting directories that contain boundary data (“master-plan-2014-planning-area-boundary-web” and “master-plan-2014-subzone-boundary-no-sea”), which provide spatial context for the POI data.
• The presence of zipped files (in KML and shapefile formats) and metadata indicates that geospatial boundary data is a significant input to the process.
Key Components
a. Data Procurement / Extraction:
– List separate modules for each POI extractor (Google, HERE, OneMap, OSM, SLA).
– Indicate that they ingest raw POI data from different external sources.
b. Schema Standardisation:
– Although there might not be a dedicated module file for this, you may group preprocessing functionality from helper_functions.py or integrate it as part of the pipeline for preparing data.
c. Taxonomy Mapping:
– Represent the taxonomy_mapper.py as a key component that aligns different POI attribute schemas.
d. POI Matching:
e. POI Unification:
– Either part of the matching module or represented as a separate step, show the function that unifies the identified duplicate or matching POI entries into a single, consolidated record.
f. Data Verification:
– Although not a specific file, include the final validation stage of the pipeline to check data quality.
g. Supporting Geospatial Boundary Data:
Architectural Relationships and Interactions
• Each extractor outputs raw POI data (these outputs may be standardized or normalized immediately).
• Feed all raw data into a “Schema Standardisation” layer (which may be a collection of helper functions).
• The standardized data is then routed to the “Taxonomy Mapping” component to harmonize the underlying classification.
• Once taxonomy mapping is complete, all datasets are combined and forwarded into the “POI Matching & Unification” module, which effectively identifies duplicates and creates a unified POI dataset.
• Finally, include a verification/check component (which can be included as part of the matching module or represented separately) to ensure data quality.
• You may also illustrate how the geospatial boundary files are used (for example, as a spatial filter, or to tag POIs with planning area metadata).
Architectural Patterns and Design Principles
• Modularity: Each major step (extraction, standardisation, mapping, matching) is in its own module.
• Pipeline/Data Flow Architecture: Data flows sequentially through each processing stage.
• Separation of Concerns: Different POI data sources are handled by dedicated extractor modules.
• Extensibility: The design allows for additional data sources to be integrated into the pipeline with minimal disruption.