# keplergl/kepler.gl-data architecture

> Architecture diagram of the GitHub repository keplergl/kepler.gl-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/keplergl/kepler.gl-data
- Repository: https://github.com/keplergl/kepler.gl-data

## Overview

##### Project Type and Purpose
- This repo is an **open-source data repository** that mirrors Uber’s kepler.gl sample data.

##### File Structure Analysis
- Top-level files:
- `.gitignore`, `LICENSE`, `README.md`, `samples.json`.
- Per-dataset directories (e.g. `earthquakes`, `nyctrips`, `sfcontour`, etc.), each containing:
- `config.json` (tells client how to interpret and render the data)
- No server-side code or build scripts—this is purely a **data & metadata** project.

##### Key Insights from README
- Each dataset entry in the README explains:
- Data source URL(s)
- File format
- The repo is consumed by kepler.gl (or similar), which reads `config.json` + data files (or remote endpoints) to render maps.

##### Main Components
a. **Dataset Repository (Git)**
– Holds multiple dataset folders.
– Versioned storage of data + config.
b. **Local File System**
– `config.json` (dataset metadata)
– `data.*` files (static assets)
– `remote-dataset-config.json` (for datasets loaded from external URLs)
c. **Remote Data Sources**
– Public APIs or web servers (USGS, OpenSky, DataSF, etc.) referenced by some `remote-dataset-config.json`.
d. **Client / Consumer**
– kepler.gl front-end (JavaScript) or any GIS/visualization client that:
- Reads `config.json`
- Fetches local data files or makes HTTP requests to remote sources
- Parses and renders geospatial data layers
e. **Optional Extensions**
– CI/CD (if any automated data validation or deploy) – not present here but could be added.
– Sample manifest (`samples.json`) listing available datasets.

##### Relationships & Data Flow
1. **Clone or Download** the repo → local filesystem.
2. Client reads `samples.json` or directory listing → user selects a dataset.
3. Client loads `config.json` for that dataset → determines file format, schema, style preferences.
4. Based on config:
- If `data.*` is local → read from disk.
- If `remote-dataset-config.json` → issue HTTP GET to remote URL(s).
5. Client parses data → constructs in-memory layers → renders map.

##### Architectural Patterns & Design Principles
- **Separation of Concerns**: Data vs. Metadata (`data.*` vs. `config.json`).
- **Plugin-style Extensibility**: Add a new dataset by dropping a folder with `config.json` + data.
- **Data Abstraction**: Client need not know detail of each file format—config drives parsing logic.
- **Static-First with Optional Remote Fetch**: Default to local files, but remote sources supported.

##### Putting It All Together
- Layer 1 (left): **Git Repository** containing multiple dataset folders.
- Layer 2 (center): **Local File System** nodes:
- Dataset n
– config.json
– data file(s) or remote-config
- Layer 3 (right): **kepler.gl Client**
- Reads config → fetches data → renders map

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: Git Repository
    subgraph "Git Repository"
        direction TB
        GitRepo["Git Repo / Dataset Folders"]:::repo
    end

    GitRepo -->|clone repo| LocalFS

    %% Layer 2: Local File System
    subgraph "Local File System"
        direction TB
        samples["samples.json"]:::meta
        subgraph "Dataset Folders"
            direction TB
            eq["earthquakes"]:::folder
            nycT["nyctrips"]:::folder
            sfC["sfcontour"]:::folder
            sft["sftrees"]:::folder
            countyU["county_unemployment"]:::folder
            laA["la_assessorparcels"]:::folder
            mvP["movement_pittsburgh"]:::folder
            nycC["nyc_census"]:::folder
            nycE["nyc_earnings"]:::folder
            ukc["ukcommute"]:::folder
            wf["world_flights"]:::folder
            usC["us_county"]:::folder
            usd["usdot"]:::folder
            llr["lake_lucerne_raster"]:::folder
            fsq["fsq_places"]:::folder
            mvt["mvt_population"]:::folder
            wms["wms_conus_cref_qcd"]:::folder
        end
    end

    LocalFS["Local File System"]:::filesystem -->|load manifest| samples
    LocalFS -->|list datasets| eq
    LocalFS --> nycT
    LocalFS --> sfC
    LocalFS --> sft
    LocalFS --> countyU
    LocalFS --> laA
    LocalFS --> mvP
    LocalFS --> nycC
    LocalFS --> nycE
    LocalFS --> ukc
    LocalFS --> wf
    LocalFS --> usC
    LocalFS --> usd
    LocalFS --> llr
    LocalFS --> fsq
    LocalFS --> mvt
    LocalFS --> wms

    samples -->|"select dataset"| Client

    %% Layer 3: Client
    Client["kepler.gl Client"]:::client

    eq -->|load config & data| Client
    nycT -->|load config & data| Client
    sfC -->|load config & data| Client
    sft -->|load config & data| Client
    countyU -->|load config & data| Client
    laA -->|load config & data| Client
    mvP -->|load config & data| Client
    nycC -->|load config & data| Client
    nycE -->|load config & data| Client
    ukc -->|load config & data| Client
    wf -->|load config & data| Client
    usC -->|load config & data| Client
    usd -->|load config & data| Client

    llr -->|remote-config| Client
    fsq -->|remote-config| Client
    mvt -->|remote-config| Client
    wms -->|remote-config| Client

    %% Layer 4: External Data Sources
    External["External Data Sources"]:::remote
    Client -->|HTTP GET remote data| External

    %% Click Events
    click GitRepo "https://github.com/keplergl/kepler.gl-data/tree/master//"
    click samples "https://github.com/keplergl/kepler.gl-data/blob/master//samples.json"
    click eq "https://github.com/keplergl/kepler.gl-data/tree/master//earthquakes"
    click nycT "https://github.com/keplergl/kepler.gl-data/tree/master//nyctrips"
    click sfC "https://github.com/keplergl/kepler.gl-data/tree/master//sfcontour"
    click sft "https://github.com/keplergl/kepler.gl-data/tree/master//sftrees"
    click countyU "https://github.com/keplergl/kepler.gl-data/tree/master//county_unemployment"
    click laA "https://github.com/keplergl/kepler.gl-data/tree/master//la_assessorparcels"
    click mvP "https://github.com/keplergl/kepler.gl-data/tree/master//movement_pittsburgh"
    click nycC "https://github.com/keplergl/kepler.gl-data/tree/master//nyc_census"
    click nycE "https://github.com/keplergl/kepler.gl-data/tree/master//nyc_earnings"
    click ukc "https://github.com/keplergl/kepler.gl-data/tree/master//ukcommute"
    click wf "https://github.com/keplergl/kepler.gl-data/tree/master//world_flights"
    click usC "https://github.com/keplergl/kepler.gl-data/tree/master//us_county"
    click usd "https://github.com/keplergl/kepler.gl-data/tree/master//usdot"
    click llr "https://github.com/keplergl/kepler.gl-data/tree/master//lake_lucerne_raster"
    click fsq "https://github.com/keplergl/kepler.gl-data/tree/master//fsq_places"
    click mvt "https://github.com/keplergl/kepler.gl-data/tree/master//mvt_population"
    click wms "https://github.com/keplergl/kepler.gl-data/tree/master//wms_conus_cref_qcd"

    %% Styles
    classDef repo fill:#f2f2f2,stroke:#333,stroke-width:1px
    classDef filesystem fill:#e6f7ff,stroke:#1f78b4,stroke-width:1px
    classDef folder fill:#ffffff,stroke:#999,stroke-width:1px
    classDef meta fill:#fff2cc,stroke:#d4a017,stroke-width:1px
    classDef client fill:#e6ffe6,stroke:#33a02c,stroke-width:1px
    classDef remote fill:#e6e6ff,stroke:#6a3d9a,stroke-width:1px
```
