# onthegomap/planetiler architecture

> Architecture diagram of the GitHub repository onthegomap/planetiler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/onthegomap/planetiler
- Repository: https://github.com/onthegomap/planetiler

## Overview

##### Main Components (Top Level)

a) Input Sources Layer:
- Different data source handlers:
- OpenStreetMap (.osm.pbf)
- Natural Earth
- GeoPackage
- Shapefile
- Parquet
- To a common "Source Feature Processor" interface

b) Core Processing Layer:
- Feature Collection Pipeline
- Feature Processing Engine
- Geometry Processing Pipeline
- Feature Merge System
- Tile Coordinate System

c) Output Generation Layer:
- Tile Generation Engine
- Archive Writers (MBTiles/PMTiles)
- Compression Systems

##### Key Relationships and Data Flow

**Input Sources → Feature Processing → Geometry Processing → Tile Generation → Output Archives**

##### Important Subsystems

a) Feature Processing:
- Profile System (for customizing mapping rules)
- Expression Engine (for YAML config processing)
- Tag Processing
- Feature Collection

b) Geometry Processing:
- Geometry Simplification
- Line Splitting
- Polygon Processing
- Coordinate Transformation

c) Memory Management:
- Memory-Mapped File Handling
- Storage Management
- Cache Systems

##### Worker Pipeline

Parallel processing architecture:
- Worker Distribution System
- Task Queue Management
- Feature Sorting
- Merge Operations

##### Specific Implementation Details

- JTS Topology Integration
- Expression Engine (CEL)
- Storage Backends (MBTiles/PMTiles)
- Profile Implementation

##### Performance Components

Systems for:
- Parallel Processing
- Memory Management
- File I/O Optimization
- Resource Monitoring

##### External Integrations

Connections to:
- Prometheus Metrics
- Logging Systems
- External Data Sources
- Tile Servers

##### Multiple input/output format support

Key Technical Considerations:

- Annotations for critical performance points
- Scaling capabilities

Remember to:

- Related components visually
- Brief annotations for complex parts
- Both data and control flow

This design reflects Planetiler's architecture as a high-performance, memory-efficient vector tile generator that can process planet-scale data on a single machine.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Define styles
    classDef input fill:#90EE90,stroke:#333,stroke-width:2px
    classDef core fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef output fill:#FFA500,stroke:#333,stroke-width:2px
    classDef support fill:#D3D3D3,stroke:#333,stroke-width:2px

    %% Input Sources Layer
    subgraph InputLayer["Input Sources Layer"]
        OSM["OSM Reader (.osm.pbf)"]:::input
        NE["Natural Earth Reader"]:::input
        GP["GeoPackage Reader"]:::input
        SHP["Shapefile Reader"]:::input
        PRQ["Parquet Reader"]:::input
    end

    %% Core Processing Layer
    subgraph CoreLayer["Core Processing Layer"]
        FCP["Feature Collection Pipeline"]:::core
        FPE["Feature Processing Engine"]:::core
        GPP["Geometry Processing Pipeline"]:::core
        FMS["Feature Merge System"]:::core
        TCS["Tile Coordinate System"]:::core
    end

    %% Worker Pipeline
    subgraph WorkerLayer["Worker Pipeline"]
        WDS["Worker Distribution System"]:::core
        TQM["Task Queue Management"]:::core
        FS["Feature Sorting"]:::core
        MO["Merge Operations"]:::core
    end

    %% Memory Management
    subgraph MemoryLayer["Memory Management"]
        MMF["Memory-Mapped Files"]:::support
        SM["Storage Management"]:::support
        CS["Cache Systems"]:::support
    end

    %% Output Generation Layer
    subgraph OutputLayer["Output Generation Layer"]
        TGE["Tile Generation Engine"]:::output
        MB["MBTiles Writer"]:::output
        PM["PMTiles Writer"]:::output
        AW["Archive Writers"]:::output
    end

    %% External Integrations
    subgraph ExternalLayer["External Integrations"]
        PM_M["Prometheus Metrics"]:::support
        LOG["Logging Systems"]:::support
        EDS["External Data Sources"]:::support
        TS["Tile Servers"]:::support
    end

    %% Connections
    OSM & NE & GP & SHP & PRQ --> FCP
    FCP --> FPE
    FPE --> GPP
    GPP --> FMS
    FMS --> TCS
    TCS --> TGE
    TGE --> MB & PM & AW

    WDS --> FPE
    TQM --> WDS
    FS --> FMS
    MO --> FMS

    MMF --> SM
    SM --> CS
    CS --> FPE

    TGE --> PM_M
    TGE --> LOG
    EDS --> FCP
    AW --> TS

    %% Click events
    click OSM "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/reader/osm/OsmReader.java"
    click NE "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/reader/NaturalEarthReader.java"
    click GP "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/reader/GeoPackageReader.java"
    click SHP "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/reader/ShapefileReader.java"
    click PRQ "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/reader/parquet/ParquetReader.java"
    click FCP "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/FeatureCollector.java"
    click FPE "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/FeatureProcessor.java"
    click GPP "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/geo/GeometryPipeline.java"
    click FMS "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/FeatureMerge.java"
    click TCS "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/geo/TileCoord.java"
    click MB "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/mbtiles/Mbtiles.java"
    click PM "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/pmtiles/WriteablePmtiles.java"
    click AW "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/archive/TileArchiveWriter.java"
    click WDS "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/worker/Distributor.java"
    click TQM "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/worker/WorkQueue.java"
    click MMF "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/collection/AppendStoreMmap.java"
    click SM "https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/collection/Storage.java"
```
