How datakind/permanent-colandr-back works
Project Type and Purpose
• This is a full-stack web application (“Colandr”) for managing and automating systematic reviews.
• It exposes a RESTful API (Flask/Connexion), a static HTML/JS frontend, asynchronous tasks (Celery), and a separate Scala microservice (“pdfestrian”) for PDF extraction.
• Data is persisted in a relational database (via SQLAlchemy/Alembic migrations), and background tasks are queued via a broker (e.g., Redis or RabbitMQ).
Out the Top-Level Components
a. Frontend
– HTML_files/… holds the static client: HTML, CSS, JS.
– Served either via a CDN/static-file server or directly by the Flask app’s static endpoints.
b. API Server (Colandr)
– manage.py + Gunicorn (gunicorn_config.py) run a Flask/Connexion application.
– colandr/api/… defines resources (controllers), authentication, Swagger docs.
– colandr/models.py + colandr/lib/… contain the domain models and business logic.
– colandr/tasks.py + celery_worker.py define background jobs.
c. Task Broker
– A message broker (Redis or RabbitMQ) sits between the API server and Celery workers.
d. Celery Workers
– celery_worker.py runs one or more worker processes that consume jobs (e.g., full-text upload, data extraction).
e. PDF Processing Microservice (pdfestrian)
– A standalone Scala/Maven project under pdfestrian/.
– Exposes its own HTTP API (org/datakind/ci/pdfestrian/api/APIService).
– Performs PDF → text/document conversion, NLP feature extraction, and writes results back to storage or returns via REST.
f. Database
– PostgreSQL (or similar) managed via SQLAlchemy and Alembic migrations (migrations/… folder).
– Stores users, reviews, citations, extracted data, etc.
g. File Storage
– PDFs and RIS/BibTeX imports live on disk (colandr_data/, file system) or could be S3 in production.
Architectural Patterns
• Layered architecture:
– Presentation (static HTML/JS)
– API/Controller (Flask/Connexion)
– Service/Business Logic (colandr/lib, Celery tasks)
– Data Access (SQLAlchemy models + Migrations)
• Asynchronous background processing (Celery + Broker).
• Microservice for heavy batch processing (pdfestrian).
• Configuration via environment (.env.example, gunicorn_config.py).
• CI/CD via GitHub Actions (/.github/workflows/heroku.yml) for deployments.
Final Checklist
– Include each major directory/module as a component.
– Show interactions: user → frontend → API → DB or Broker → Worker → PDF.
– Annotate technology choices (Flask, Celery, SQLAlchemy, Scala/Maven).