# worldfishcenter/peskas.validation architecture

> Architecture diagram of the GitHub repository worldfishcenter/peskas.validation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/worldfishcenter/peskas.validation
- Repository: https://github.com/worldfishcenter/peskas.validation

## Overview

##### Project Type and Purpose
- Full-stack web portal for validating KoboToolbox survey submissions, tracking enumerator performance, and managing surveys/users.
- Frontend in React/TypeScript (Vite), backend as Express + Vercel Serverless Functions, MongoDB as primary datastore, external integrations (KoboToolbox, Airtable), and R scripts for data ingestion.

##### File Structure Analysis
- Top-level “src/” (frontend React app), “server/” (Express server for dev/production), “api/” (Vercel serverless functions), “lib/” (shared utilities used by serverless/API), “scripts/” (management/migration), configuration files (.env, tsconfig, Vercel, ESLint).
- Patterns:
- frontend/backend separation
- serverless functions in “api/” handling auth, CRUD, syncing
- Express server in “server/” for standalone deployments
- Shared utilities (db, middleware, JWT) in “lib/”
- Management scripts for Airtable sync, initial seeding, R pipeline orchestrated separately

##### README Insights
- Tech stack: React 18 + TypeScript, Express.js, MongoDB, Vercel Functions.
- Data flow:
KoboToolbox → R Pipeline → MongoDB ←→ Portal → Validation updates → KoboToolbox
- Role-based access, multi-survey support, performance dashboard, Airtable sync.
- MongoDB collections per survey and precomputed stats collections.

##### Main Components
a. Frontend (Browser)
- React/Vite app, Tabler UI, TanStack Table, Highcharts
- Routes: login, validation table, performance dashboard, admin panels
b. API Layer
- Vercel Serverless Functions (“api/” folder) for auth, users, surveys, submissions, stats, countries
- Express.js server (“server/”) for local dev or standalone deployment
- Shared utilities in “lib/” for DB connection, JWT, middleware
c. Database
- MongoDB Atlas (or self-hosted)
- Collections: users, surveys, countries, surveys_flags-{asset_id}, enumerators_stats-{asset_id}
d. R Pipeline
- R scripts fetch data from KoboToolbox, compute alerts, write to MongoDB
e. External Services
- KoboToolbox API
- Airtable API (for user & survey sync)
f. Management Scripts
- Node.js scripts under “scripts/” for seeding, migrations, sync with Airtable
g. Clients & Deployment
- Developer machine: npm run dev (front + back)
- Vercel: deploy serverless API + static frontend

##### Architectural Patterns & Principles
- Separation of concerns: distinct frontend, API, data ingestion layers
- Serverless functions for horizontal scalability, Express for local dev
- Database-first design: MongoDB as single source of truth
- Role-based access control at API endpoints
- Multi-tenant/survey pattern: dynamic collections per survey asset ID
- Eventual consistency: validation updates sync back to KoboToolbox asynchronously

##### Technologies & Frameworks
- Frontend: React 18, TypeScript, Vite, React Router v7, TanStack Table, Highcharts, Tabler Core
- Backend: Node.js, Express.js, Vercel Serverless Functions, JWT, middleware
- Database: MongoDB (Atlas or self-hosted)
- Data Ingestion: R (scripts), KoboToolbox API
- Integrations: Airtable API

##### Essential Detail
- External integrations: KoboToolbox, Airtable, email/JWT
- Both real-time API calls during page loads and asynchronous ingestion/update flows
- Don’t omit the R Pipeline or management scripts—they are critical for data preprocessing and sync

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer" 
        direction TB
        ReactApp["React App (Vite)"]:::frontend
        MainTS["main.tsx"]:::frontend
        AppTS["App.tsx"]:::frontend
        Components["Components (UI)"]:::frontend
        APIClient["API Client Utils"]:::frontend
        Utils["Utils"]:::frontend
        ReactApp --> MainTS
        ReactApp --> AppTS
        ReactApp --> Components
        ReactApp --> APIClient
        ReactApp --> Utils
    end

    %% API Layer
    subgraph "API Layer (Vercel Functions)" 
        direction TB
        Auth["Auth Functions"]:::api
        Users["Users Functions"]:::api
        Surveys["Surveys Functions"]:::api
        Submissions["Submissions & Validation"]:::api
        Countries["Countries Functions"]:::api
        Stats["Enumerator Stats"]:::api
        AdminScripts["Admin Scripts"]:::api
        Auth --> Users
        Users --> Surveys
        Surveys --> Submissions
        Submissions --> Countries
        Countries --> Stats
        Stats --> AdminScripts
    end

    %% Express Server for Dev/Standalone
    subgraph "Express Server" 
        direction TB
        DevServer["dev.js"]:::api
        ProdServer["index.js"]:::api
        DevServer --> ProdServer
    end

    %% Shared Utilities
    subgraph "Shared Utilities (lib)" 
        direction TB
        DBConn["db.js"]:::api
        JWT["jwt.js"]:::api
        Middleware["middleware.js"]:::api
        APIUtils["api-utils.js"]:::api
        Response["response.js"]:::api
        Helpers["helpers.js"]:::api
        DBConn --> JWT
        JWT --> Middleware
        Middleware --> APIUtils
        APIUtils --> Response
        Response --> Helpers
    end

    %% Database
    MongoDB["MongoDB Atlas"]:::db

    %% R Pipeline
    RPipeline["R Pipeline Scripts"]:::script

    %% Management Scripts
    Scripts["Management Scripts"]:::script

    %% External Services
    Kobo["KoboToolbox API"]:::ext
    Airtable["Airtable API"]:::ext
    EmailAuth["Email/JWT Auth Service"]:::ext

    %% Data Flows
    Browser["User Browser"]:::frontend
    Browser -->|load| ReactApp
    ReactApp -->|HTTP/REST| Auth
    ReactApp -->|HTTP/REST| Users
    ReactApp -->|HTTP/REST| Surveys
    ReactApp -->|HTTP/REST| Submissions
    ReactApp -->|HTTP/REST| Countries
    ReactApp -->|HTTP/REST| Stats

    Auth -->|DB read/write| MongoDB
    Users -->|DB read/write| MongoDB
    Surveys -->|DB read/write| MongoDB
    Submissions -->|DB read/write| MongoDB
    Countries -->|DB read/write| MongoDB
    Stats -->|DB read/write| MongoDB

    RPipeline -->|write submissions & stats| MongoDB
    RPipeline -->|fetch data| Kobo
    AdminScripts -->|sync users & stats| Airtable
    AdminScripts -->|DB update| MongoDB
    Submissions -->|on validation update| Kobo
    Auth -->|external auth| EmailAuth

    %% Click Events - Frontend
    click ReactApp "https://github.com/worldfishcenter/peskas.validation/tree/main/src/"
    click MainTS "https://github.com/worldfishcenter/peskas.validation/blob/main/src/main.tsx"
    click AppTS "https://github.com/worldfishcenter/peskas.validation/blob/main/src/App.tsx"
    click Components "https://github.com/worldfishcenter/peskas.validation/tree/main/src/components/"
    click APIClient "https://github.com/worldfishcenter/peskas.validation/tree/main/src/api/"
    click Utils "https://github.com/worldfishcenter/peskas.validation/tree/main/src/utils/"

    %% Click Events - API Layer
    click Auth "https://github.com/worldfishcenter/peskas.validation/blob/main/api/auth/login.js"
    click Auth "https://github.com/worldfishcenter/peskas.validation/blob/main/api/auth/me.js"
    click Users "https://github.com/worldfishcenter/peskas.validation/blob/main/api/users/index.js"
    click Users "https://github.com/worldfishcenter/peskas.validation/blob/main/api/users/[id]/index.js"
    click Users "https://github.com/worldfishcenter/peskas.validation/blob/main/api/users/[id]/permissions.js"
    click Users "https://github.com/worldfishcenter/peskas.validation/blob/main/api/users/[id]/reset-password.js"
    click Users "https://github.com/worldfishcenter/peskas.validation/blob/main/api/users/[id]/accessible-surveys.js"
    click Surveys "https://github.com/worldfishcenter/peskas.validation/blob/main/api/surveys/index.js"
    click Surveys "https://github.com/worldfishcenter/peskas.validation/blob/main/api/surveys/[asset_id]/index.js"
    click Surveys "https://github.com/worldfishcenter/peskas.validation/blob/main/api/surveys/[asset_id]/alert-codes.js"
    click Submissions "https://github.com/worldfishcenter/peskas.validation/blob/main/api/submissions/[id]/validation-status.js"
    click Submissions "https://github.com/worldfishcenter/peskas.validation/blob/main/api/kobo/submissions.js"
    click Submissions "https://github.com/worldfishcenter/peskas.validation/blob/main/api/kobo/validation-status/index.js"
    click Submissions "https://github.com/worldfishcenter/peskas.validation/blob/main/api/kobo/validation-status/[id].js"
    click Countries "https://github.com/worldfishcenter/peskas.validation/blob/main/api/countries/index.js"
    click Countries "https://github.com/worldfishcenter/peskas.validation/blob/main/api/countries/[code].js"
    click Stats "https://github.com/worldfishcenter/peskas.validation/blob/main/api/enumerators-stats.js"
    click AdminScripts "https://github.com/worldfishcenter/peskas.validation/blob/main/api/admin/sync-users.js"
    click AdminScripts "https://github.com/worldfishcenter/peskas.validation/blob/main/api/admin/refresh-enumerator-stats.js"

    %% Click Events - Express Server
    click DevServer "https://github.com/worldfishcenter/peskas.validation/blob/main/server/dev.js"
    click ProdServer "https://github.com/worldfishcenter/peskas.validation/blob/main/server/index.js"

    %% Click Events - Shared Utilities
    click DBConn "https://github.com/worldfishcenter/peskas.validation/blob/main/lib/db.js"
    click JWT "https://github.com/worldfishcenter/peskas.validation/blob/main/lib/jwt.js"
    click Middleware "https://github.com/worldfishcenter/peskas.validation/blob/main/lib/middleware.js"
    click APIUtils "https://github.com/worldfishcenter/peskas.validation/blob/main/lib/api-utils.js"
    click Response "https://github.com/worldfishcenter/peskas.validation/blob/main/lib/response.js"
    click Helpers "https://github.com/worldfishcenter/peskas.validation/blob/main/lib/helpers.js"

    classDef frontend fill:#E0F7FA,stroke:#0288D1,color:#000
    classDef api fill:#C8E6C9,stroke:#2E7D32,color:#000
    classDef db fill:#FFCC80,stroke:#EF6C00,color:#000
    classDef script fill:#B0BEC5,stroke:#546E7A,color:#000
    classDef ext fill:#E1BEE7,stroke:#6A1B9A,color:#000
```
