How keplergl/kepler.gl-data works
Project Type and Purpose
• This repo is an open-source data repository that mirrors Uber’s kepler.gl sample data.
File Structure Analysis
• Top-level files:
• .gitignore, LICENSE, README.md, samples.json.
• Per-dataset directories (e.g. earthquakes, nyctrips, sfcontour, etc.), each containing:
• config.json (tells client how to interpret and render the data)
• No server-side code or build scripts—this is purely a data & metadata project.
Key Insights from README
• Each dataset entry in the README explains:
• Data source URL(s)
• File format
• The repo is consumed by kepler.gl (or similar), which reads config.json + data files (or remote endpoints) to render maps.
Main Components
a. Dataset Repository (Git)
– Holds multiple dataset folders.
– Versioned storage of data + config.
b. Local File System
– config.json (dataset metadata)
– data.* files (static assets)
– remote-dataset-config.json (for datasets loaded from external URLs)
c. Remote Data Sources
– Public APIs or web servers (USGS, OpenSky, DataSF, etc.) referenced by some remote-dataset-config.json.
d. Client / Consumer
– kepler.gl front-end (JavaScript) or any GIS/visualization client that:
• Reads config.json
• Fetches local data files or makes HTTP requests to remote sources
• Parses and renders geospatial data layers
e. Optional Extensions
– CI/CD (if any automated data validation or deploy) – not present here but could be added.
– Sample manifest (samples.json) listing available datasets.
Relationships & Data Flow
1. Clone or Download the repo → local filesystem.
2. Client reads samples.json or directory listing → user selects a dataset.
3. Client loads config.json for that dataset → determines file format, schema, style preferences.
4. Based on config:
• If data.* is local → read from disk.
• If remote-dataset-config.json → issue HTTP GET to remote URL(s).
5. Client parses data → constructs in-memory layers → renders map.
Architectural Patterns & Design Principles
• Separation of Concerns: Data vs. Metadata (data.* vs. config.json).
• Plugin-style Extensibility: Add a new dataset by dropping a folder with config.json + data.
• Data Abstraction: Client need not know detail of each file format—config drives parsing logic.
• Static-First with Optional Remote Fetch: Default to local files, but remote sources supported.
Putting It All Together
• Layer 1 (left): Git Repository containing multiple dataset folders.
• Layer 2 (center): Local File System nodes:
• Dataset n
– config.json
– data file(s) or remote-config
• Layer 3 (right): kepler.gl Client
• Reads config → fetches data → renders map