How paristarbel/paleo-vault works
PaleoVault is an ETL and exploration application for South African fossil occurrences. The principal workflow fetches fossil records from GBIF, validates and flags coordinates and missing fields, exports cleaned records, loads them into PostgreSQL, and serves SQL-backed charts, maps, and species exploration through Streamlit. The species branch additionally retrieves specimen media from stored JSON and optional Wikipedia summaries. The graph omits unsampled deployment infrastructure and tests; the generated CSV is represented by its producing module because it is absent from the tree.
Components 16
Ingestion Quality
- Extract Clean
gbif_API.py - Clean Records
gbif_API.py
Data Persistence
- Database Loader
load_to_db.py - Records Database
load_to_db.py
Analysis Enrichment
- SQL Query Layer
queries.py - Aggregate Analysis
queries.py - Species Analysis
queries.py - Media Lookup
queries.py - Wikipedia Lookup
queries.py
User Experience
- Streamlit Dashboard
dashboard.py - Trend Charts
dashboard.py - Fossil Maps
dashboard.py - Species Explorer
dashboard.py
Other components
- Researcher
- GBIF API
- Wikipedia API
Connections 16
- Extract Clean → GBIF API: fetches records
- Extract Clean → Clean Records: writes records
- Clean Records → Database Loader: supplies CSV
- Database Loader → Records Database: inserts rows
- Streamlit Dashboard → SQL Query Layer: requests data
- SQL Query Layer → Records Database: runs SQL
- SQL Query Layer → Aggregate Analysis: returns aggregates
- SQL Query Layer → Species Analysis: returns species data
- SQL Query Layer → Media Lookup: reads media
- Media Lookup → Records Database: queries media
- SQL Query Layer → Wikipedia Lookup: requests profile
- Wikipedia Lookup → Wikipedia API: fetches summary
- Streamlit Dashboard → Trend Charts: renders charts
- Streamlit Dashboard → Fossil Maps: renders maps
- Streamlit Dashboard → Species Explorer: renders explorer
- Researcher → Streamlit Dashboard: explores data