How worldfishcenter/peskas.validation works
Project Type and Purpose
• Full-stack web portal for validating KoboToolbox survey submissions, tracking enumerator performance, and managing surveys/users.
• Frontend in React/TypeScript (Vite), backend as Express + Vercel Serverless Functions, MongoDB as primary datastore, external integrations (KoboToolbox, Airtable), and R scripts for data ingestion.
File Structure Analysis
• Top-level “src/” (frontend React app), “server/” (Express server for dev/production), “api/” (Vercel serverless functions), “lib/” (shared utilities used by serverless/API), “scripts/” (management/migration), configuration files (.env, tsconfig, Vercel, ESLint).
• Patterns:
• frontend/backend separation
• serverless functions in “api/” handling auth, CRUD, syncing
• Express server in “server/” for standalone deployments
• Shared utilities (db, middleware, JWT) in “lib/”
• Management scripts for Airtable sync, initial seeding, R pipeline orchestrated separately
README Insights
• Tech stack: React 18 + TypeScript, Express.js, MongoDB, Vercel Functions.
• Data flow:
KoboToolbox → R Pipeline → MongoDB ←→ Portal → Validation updates → KoboToolbox
• Role-based access, multi-survey support, performance dashboard, Airtable sync.
• MongoDB collections per survey and precomputed stats collections.
Main Components
a. Frontend (Browser)
• React/Vite app, Tabler UI, TanStack Table, Highcharts
• Routes: login, validation table, performance dashboard, admin panels
b. API Layer
• Vercel Serverless Functions (“api/” folder) for auth, users, surveys, submissions, stats, countries
• Express.js server (“server/”) for local dev or standalone deployment
• Shared utilities in “lib/” for DB connection, JWT, middleware
c. Database
• MongoDB Atlas (or self-hosted)
• Collections: users, surveys, countries, surveys_flags-{asset_id}, enumerators_stats-{asset_id}
d. R Pipeline
• R scripts fetch data from KoboToolbox, compute alerts, write to MongoDB
e. External Services
• KoboToolbox API
• Airtable API (for user & survey sync)
f. Management Scripts
• Node.js scripts under “scripts/” for seeding, migrations, sync with Airtable
g. Clients & Deployment
• Developer machine: npm run dev (front + back)
• Vercel: deploy serverless API + static frontend
Architectural Patterns & Principles
• Separation of concerns: distinct frontend, API, data ingestion layers
• Serverless functions for horizontal scalability, Express for local dev
• Database-first design: MongoDB as single source of truth
• Role-based access control at API endpoints
• Multi-tenant/survey pattern: dynamic collections per survey asset ID
• Eventual consistency: validation updates sync back to KoboToolbox asynchronously
Technologies & Frameworks
• Frontend: React 18, TypeScript, Vite, React Router v7, TanStack Table, Highcharts, Tabler Core
• Backend: Node.js, Express.js, Vercel Serverless Functions, JWT, middleware
• Database: MongoDB (Atlas or self-hosted)
• Data Ingestion: R (scripts), KoboToolbox API
• Integrations: Airtable API
Essential Detail
• External integrations: KoboToolbox, Airtable, email/JWT
• Both real-time API calls during page loads and asynchronous ingestion/update flows
• Don’t omit the R Pipeline or management scripts—they are critical for data preprocessing and sync