# dali-lab/pine-beetle-automation architecture

> Architecture diagram of the GitHub repository dali-lab/pine-beetle-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dali-lab/pine-beetle-automation
- Repository: https://github.com/dali-lab/pine-beetle-automation

## Overview

##### Project Type & Purpose
- This is a Node/Express “automation” microservice in the Project Pine Beetle ecosystem.
- Purpose: ingest raw outbreak data via Survey123 webhooks, transform & aggregate it, run predictive models in R, store results in MongoDB, and expose HTTP endpoints for downstream systems (main backend or frontend).

##### File Structure
- Top-level config & CI:
– .babelrc, .eslintrc (language tooling)
– .github/workflows/nodejs.ci.yml (CI/CD)
– Procfile (Heroku deployment)
- src/index.js: Express server bootstrap
- src/routers/: HTTP route definitions (e.g. /pipeline, /survey123, /summarized-county, etc.)
- src/middleware/: reusable Express middleware (e.g. require-auth)
- src/controllers/: business-logic layer—validates inputs, orchestrates data flows, wraps aggregation & R predictions
- src/models/: Mongoose schemas for each data type (summarized-county, unsummarized-trapping, etc.)
- src/utils/: shared helpers—CSV parsing, Mongo connection, R-process launcher, response formatting, validators
- src/constants/: JSON mapping files (state abbreviations, response-codes/types)
- src/r-scripts/: R scripts implementing the statistical predictive model and calculated fields
- docs/: detailed API documentation and process guides (ROUTES.md, pipeline.md, etc.)
- test/: verification script for predictions

##### Key Insights from the README
- This repo works alongside a “main backend” that handles CRUD and auth.
- Data enters via Survey123 webhooks (ArcGIS), optionally through ngrok locally.
- Predictive engine is written in R (launched via r-launcher util).
- MongoDB stores both raw and aggregated JSON data.
- Deployment is on Heroku with GitHub Actions–powered CI/CD.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Sources
    Survey123(("Survey123 Webhooks")):::external
    Ngrok(("ngrok Tunnel")):::external

    %% Application Entry Point
    Express["Express Server\n(src/index.js)"]:::api
    click Express "https://github.com/dali-lab/pine-beetle-automation/blob/dev/src/index.js"

    %% API Layer
    subgraph "API Layer"
        Routers["Routers\n(src/routers/)"]:::api
        click Routers "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/routers/"
    end

    %% Middleware Layer
    subgraph "Middleware"
        Middleware["Auth & Shared Middleware\n(src/middleware/)"]:::api
        click Middleware "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/middleware/"
    end

    %% Business Logic Layer
    subgraph "Business Logic"
        Controllers["Controllers\n(src/controllers/)"]:::business
        click Controllers "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/controllers/"
    end

    %% Utility Libraries
    subgraph "Shared Utilities"
        Utils["Utils\n(src/utils/)"]:::utility
        click Utils "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/utils/"
    end

    %% Predictive Engine
    Predictive["Predictive Engine (R)\n(src/r-scripts/)"]:::business
    click Predictive "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/r-scripts/"

    %% Data Layer
    Mongo[(MongoDB)]:::database
    Models["Mongoose Models\n(src/models/)"]:::database
    click Models "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/models/"
    Constants["Constants & Mappings\n(src/constants/)"]:::database
    click Constants "https://github.com/dali-lab/pine-beetle-automation/tree/dev/src/constants/"

    %% Consumers & Support
    Consumer(("Main Backend / Frontend")):::external
    CI["CI/CD\n(.github/workflows/nodejs.ci.yml)"]:::external
    click CI "https://github.com/dali-lab/pine-beetle-automation/blob/dev/.github/workflows/nodejs.ci.yml"
    Docs["Documentation\n(docs/)"]:::external
    click Docs "https://github.com/dali-lab/pine-beetle-automation/tree/dev/docs/"
    Tests["Tests & Verification\n(test/)"]:::external
    click Tests "https://github.com/dali-lab/pine-beetle-automation/tree/dev/test/"

    %% Data Flows
    Survey123 -->|Webhook POST| Ngrok -->|HTTP 9091| Express
    Express --> Routers --> Middleware --> Controllers
    Controllers --> Utils
    Controllers --> Predictive
    Controllers --> Models --> Mongo
    Controllers --> Constants
    Controllers -->|JSON Response| Consumer
    CI -->|Deploy| Express

    %% Styles
    classDef external fill:#DDDDDD,stroke:#888;
    classDef api fill:#D6EAF8,stroke:#3182bd,color:#000;
    classDef business fill:#D5F5E3,stroke:#27ae60,color:#000;
    classDef utility fill:#FCF3CF,stroke:#b7950b,color:#000;
    classDef database fill:#FAD7A0,stroke:#d35400,color:#000
```
