How dali-lab/pine-beetle-automation works
Project Type & Purpose
• This is a Node/Express “automation” microservice in the Project Pine Beetle ecosystem.
• Purpose: ingest raw outbreak data via Survey123 webhooks, transform & aggregate it, run predictive models in R, store results in MongoDB, and expose HTTP endpoints for downstream systems (main backend or frontend).
File Structure
• Top-level config & CI:
– .babelrc, .eslintrc (language tooling)
– .github/workflows/nodejs.ci.yml (CI/CD)
– Procfile (Heroku deployment)
• src/index.js: Express server bootstrap
• src/routers/: HTTP route definitions (e.g. /pipeline, /survey123, /summarized-county, etc.)
• src/middleware/: reusable Express middleware (e.g. require-auth)
• src/controllers/: business-logic layer—validates inputs, orchestrates data flows, wraps aggregation & R predictions
• src/models/: Mongoose schemas for each data type (summarized-county, unsummarized-trapping, etc.)
• src/utils/: shared helpers—CSV parsing, Mongo connection, R-process launcher, response formatting, validators
• src/constants/: JSON mapping files (state abbreviations, response-codes/types)
• src/r-scripts/: R scripts implementing the statistical predictive model and calculated fields
• docs/: detailed API documentation and process guides (ROUTES.md, pipeline.md, etc.)
• test/: verification script for predictions
Key Insights from the README
• This repo works alongside a “main backend” that handles CRUD and auth.
• Data enters via Survey123 webhooks (ArcGIS), optionally through ngrok locally.
• Predictive engine is written in R (launched via r-launcher util).
• MongoDB stores both raw and aggregated JSON data.
• Deployment is on Heroku with GitHub Actions–powered CI/CD.