# datakind/permanent-colandr-back architecture

> Architecture diagram of the GitHub repository datakind/permanent-colandr-back, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datakind/permanent-colandr-back
- Repository: https://github.com/datakind/permanent-colandr-back

## Overview

##### Project Type and Purpose
- This is a full-stack web application (“Colandr”) for managing and automating systematic reviews.
- It exposes a RESTful API (Flask/Connexion), a static HTML/JS frontend, asynchronous tasks (Celery), and a separate Scala microservice (“pdfestrian”) for PDF extraction.
- Data is persisted in a relational database (via SQLAlchemy/Alembic migrations), and background tasks are queued via a broker (e.g., Redis or RabbitMQ).

##### Out the Top-Level Components
a. Frontend
– HTML_files/… holds the static client: HTML, CSS, JS.
– Served either via a CDN/static-file server or directly by the Flask app’s static endpoints.
b. API Server (Colandr)
– manage.py + Gunicorn (gunicorn_config.py) run a Flask/Connexion application.
– colandr/api/… defines resources (controllers), authentication, Swagger docs.
– colandr/models.py + colandr/lib/… contain the domain models and business logic.
– colandr/tasks.py + celery_worker.py define background jobs.
c. Task Broker
– A message broker (Redis or RabbitMQ) sits between the API server and Celery workers.
d. Celery Workers
– celery_worker.py runs one or more worker processes that consume jobs (e.g., full-text upload, data extraction).
e. PDF Processing Microservice (pdfestrian)
– A standalone Scala/Maven project under pdfestrian/.
– Exposes its own HTTP API (org/datakind/ci/pdfestrian/api/APIService).
– Performs PDF → text/document conversion, NLP feature extraction, and writes results back to storage or returns via REST.
f. Database
– PostgreSQL (or similar) managed via SQLAlchemy and Alembic migrations (migrations/… folder).
– Stores users, reviews, citations, extracted data, etc.
g. File Storage
– PDFs and RIS/BibTeX imports live on disk (colandr_data/, file system) or could be S3 in production.

##### Architectural Patterns
- Layered architecture:
– Presentation (static HTML/JS)
– API/Controller (Flask/Connexion)
– Service/Business Logic (colandr/lib, Celery tasks)
– Data Access (SQLAlchemy models + Migrations)
- Asynchronous background processing (Celery + Broker).
- Microservice for heavy batch processing (pdfestrian).
- Configuration via environment (.env.example, gunicorn_config.py).
- CI/CD via GitHub Actions (/.github/workflows/heroku.yml) for deployments.

##### Final Checklist
– Include each major directory/module as a component.
– Show interactions: user → frontend → API → DB or Broker → Worker → PDF.
– Annotate technology choices (Flask, Celery, SQLAlchemy, Scala/Maven).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and Frontend
    User["Browser (User)"]:::frontend
    Frontend["Static Frontend\n(HTML_files/)"]:::frontend
    click Frontend "https://github.com/datakind/permanent-colandr-back/tree/main/HTML_files/"

    User -->|"HTTP GET"| Frontend

    %% API Server Layer
    subgraph "API Server (Flask + Connexion)" 
        direction TB
        Entry["manage.py"]:::api
        click Entry "https://github.com/datakind/permanent-colandr-back/blob/main/manage.py"
        GunicornConf["gunicorn_config.py"]:::api
        click GunicornConf "https://github.com/datakind/permanent-colandr-back/blob/main/gunicorn_config.py"
        GunicornRun["gunicorn_runserver.py"]:::api
        click GunicornRun "https://github.com/datakind/permanent-colandr-back/blob/main/gunicorn_runserver.py"
        Config["colandr/config.py"]:::api
        click Config "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/config.py"
        Controllers["colandr/api/resources/"]:::api
        click Controllers "https://github.com/datakind/permanent-colandr-back/tree/main/colandr/api/resources/"
        Auth["authentication.py"]:::api
        click Auth "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/api/authentication.py"
        Reg["registration.py"]:::api
        click Reg "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/api/registration.py"
        Swagger["swagger.py"]:::api
        click Swagger "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/api/swagger.py"
        Schemas["schemas.py"]:::api
        click Schemas "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/api/schemas.py"
        Utils["utils.py"]:::api
        click Utils "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/api/utils.py"
        Errors["errors.py"]:::api
        click Errors "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/api/errors.py"
        Models["colandr/models.py"]:::api
        click Models "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/models.py"
        Lib["colandr/lib/"]:::api
        click Lib "https://github.com/datakind/permanent-colandr-back/tree/main/colandr/lib/"
        TasksDef["colandr/tasks.py"]:::async
        click TasksDef "https://github.com/datakind/permanent-colandr-back/blob/main/colandr/tasks.py"
        WorkerScript["celery_worker.py"]:::async
        click WorkerScript "https://github.com/datakind/permanent-colandr-back/blob/main/celery_worker.py"
        Emails["colandr/templates/emails/"]:::api
        click Emails "https://github.com/datakind/permanent-colandr-back/tree/main/colandr/templates/emails/"
        Migrations["migrations/"]:::datastore
        click Migrations "https://github.com/datakind/permanent-colandr-back/tree/main/migrations/"
        Storage["colandr_data/"]:::datastore
        click Storage "https://github.com/datakind/permanent-colandr-back/tree/main/colandr_data/"
        Env[".env.example"]:::api
        click Env "https://github.com/datakind/permanent-colandr-back/blob/main/.env.example"
        CI[".github/workflows/heroku.yml"]:::api
        click CI "https://github.com/datakind/permanent-colandr-back/blob/main/.github/workflows/heroku.yml"
    end

    Frontend -->|"AJAX/Fetch"| Entry
    Entry -->|"invokes"| Controllers
    Controllers -->|"uses"| Auth
    Controllers -->|"uses"| Reg
    Controllers -->|"documents"| Swagger
    Controllers -->|"validates via"| Schemas
    Controllers -->|"calls"| Utils
    Controllers -->|"handles"| Errors
    Controllers -->|"loads config"| Config
    Controllers -->|"CRUD via"| Models
    Controllers -->|"business logic"| Lib
    Controllers -->|"enqueue tasks"| TasksDef

    %% Database
    DB[(PostgreSQL\n+ Alembic)]:::datastore
    Controllers -->|"sync read/write"| DB
    TasksDef -->|"sync read/write"| DB
    WorkerScript -->|"sync read/write"| DB

    %% Asynchronous Broker and Workers
    Broker((Message Broker\nRedis/RabbitMQ)):::async
    Controllers -->|"---> publish task"| Broker
    Broker -->|"---> deliver job"| WorkerScript
    WorkerScript -->|"runs tasks"| TasksDef

    %% PDF Processing Microservice
    subgraph "PDF Service (pdfestrian)" 
        direction TB
        PDFRoot["pdfestrian/"]:::external
        click PDFRoot "https://github.com/datakind/permanent-colandr-back/tree/main/pdfestrian/"
        POM["pom.xml"]:::external
        click POM "https://github.com/datakind/permanent-colandr-back/blob/main/pdfestrian/pom.xml"
        ConfigPDF["pdfestrian/config/pdfestrian_default.conf"]:::external
        click ConfigPDF "https://github.com/datakind/permanent-colandr-back/blob/main/pdfestrian/config/pdfestrian_default.conf"
        ConfigRes["src/main/resources/pdfestrian.conf"]:::external
        click ConfigRes "https://github.com/datakind/permanent-colandr-back/blob/main/pdfestrian/src/main/resources/pdfestrian.conf"
        APIImpl["APIServiceImpl.scala"]:::external
        click APIImpl "https://github.com/datakind/permanent-colandr-back/blob/main/pdfestrian/src/main/scala/org/datakind/ci/pdfestrian/api/apiservice/APIServiceImpl.scala"
        CoreExtract["extractText.scala"]:::external
        click CoreExtract "https://github.com/datakind/permanent-colandr-back/blob/main/pdfestrian/src/main/scala/org/datakind/ci/pdfestrian/pdfExtraction/extractText.scala"
        DBComp["db/"]:::external
        click DBComp "https://github.com/datakind/permanent-colandr-back/tree/main/pdfestrian/src/main/scala/org/datakind/ci/pdfestrian/db/"
        Features["features/"]:::external
        click Features "https://github.com/datakind/permanent-colandr-back/tree/main/pdfestrian/src/main/scala/org/datakind/ci/pdfestrian/extraction/features/"
        Bin["bin/"]:::external
        click Bin "https://github.com/datakind/permanent-colandr-back/tree/main/pdfestrian/bin/"
    end

    WorkerScript -->|"- - - HTTP"| APIImpl
    APIImpl -->|"process PDF"| CoreExtract
    CoreExtract -->|"store results"| DBComp
    CoreExtract -->|"extract features"| Features
    CoreExtract -->|"return text/metadata"| APIImpl
    APIImpl -->|"- - - HTTP RESPONSE"| WorkerScript

    %% File storage interactions
    WorkerScript -->|"save PDF/text"| Storage

    %% Email and Exports
    Controllers -->|"generate/send"| Emails

    %% Styles
    classDef frontend fill:#9ad1f5,stroke:#036fa0,color:#000
    classDef api fill:#b2f2b6,stroke:#0b6623,color:#000
    classDef datastore fill:#ffd8a8,stroke:#d9822b,color:#000
    classDef async fill:#eebcff,stroke:#8a2be2,color:#000
    classDef external fill:#d3d3d3,stroke:#696969,color:#000
```
