# ourresearch/openalex-keywords architecture

> Architecture diagram of the GitHub repository ourresearch/openalex-keywords, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ourresearch/openalex-keywords
- Repository: https://github.com/ourresearch/openalex-keywords

## Overview

##### Project Type and Purpose
- The repository is centered around generating keywords and exposes functionality through a model, now containerized and packaged for deployment (especially in v2).
- v1 contains notebooks (e.g., run_keywords.ipynb, store_keywords_SPARK.ipynb) that likely serve as experiments or early-stage implementations.
- v2 focuses on deployment: it provides structured folders for deployment (003_Deployment) that include the API model endpoint, Docker container configuration, and scripts for both deployment and local testing.

##### Main Components to Identify
a. Data/Model Generation Component:
- Keyword generation logic that is developed in v1 notebooks.
- This is the core algorithm or ML model that generates keywords.
b. Deployment & API Component:
- In v2/003_Deployment, particularly under “model_to_api”, you find the API server components.
- Inside “model_to_api/container”, there is a dedicated setup including:
- A Dockerfile and a build script (build_and_push.sh) for containerization.
- The “keyword_matcher” subfolder with nginx.conf (acting as a reverse proxy), predictor.py (executing the logic or inference), serve scripts (likely initiating service), and wsgi.py (a WSGI entry point for deploying Python web apps).
c. Testing & Local Execution:
- Local test scripts (e.g., v2/003_Deployment/model_to_api/local_test/predict.sh and serve_local.sh) enable testing of the API endpoint locally.
d. Integration with Third-Party/External Services:
- The README points to https://openalex.org/, so this could be an external integration or reference for dataset/keyword context.

##### Relationships and Interactions
- The container (with its components like nginx for routing and WSGI for application server) forwards the request to the underlying keyword generation logic (i.e., predictor.py).
- Data flows from the API endpoint to the model logic, and the outputs (generated keywords) are returned to the client.
- Testing components (local test scripts) mimic these interactions in a local development environment.

##### Architectural Patterns & Design Principles
- Microservices/Containerization: The project leverages Docker containers to isolate and deploy the API service.
- Separation of Concerns: There’s a clear delineation between experimental/model training (v1 notebooks) and production deployments (v2 deployment scripts).
- API Abstraction: The keyword generation logic is encapsulated behind a RESTful or WSGI-based interface, facilitating integration with other systems.
- Cloud/Nginx Proxy: Use of an nginx configuration implies a reverse proxy pattern for load balancing or security.

##### Technologies, Frameworks, and Framework-specific Elements
- Docker: Container usage is evident from the Dockerfile and build script, indicating that the project is intended to run in containerized environments.
- Python & Jupyter Notebooks: Keywords generation initially implemented via Python notebooks.
- Nginx and WSGI: For serving the API in the container.
- Shell Scripts: For deploying and testing (build_and_push.sh, predict.sh, serve_local.sh).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    CR["Client Request"]:::external

    %% Experimental Model (v1)
    subgraph "Experimental Model (v1)"
        KN1["run_keywords.ipynb"]:::experimental
        KN2["store_keywords_SPARK.ipynb"]:::experimental
    end

    %% Deployment & API (v2)
    subgraph "Deployment & API (v2)"
        DS["Docker Config & Build Scripts"]:::deployment
        DC["Docker Container"]:::deployment
        %% API Server - Keyword Matcher inside container
        subgraph "API Server – Keyword Matcher"
            N["nginx Reverse Proxy"]:::deployment
            W["WSGI Application"]:::deployment
            P["Keyword Predictor Module"]:::deployment
            S["Serve Scripts"]:::deployment
        end
        TEST["Local Testing Scripts"]:::deployment
        EXT["OpenAlex Endpoint Setup"]:::deployment
    end

    %% Request Flow
    CR -->|"HTTP_Request"| N
    N -->|"forward_request"| W
    W -->|"invoke_predictor"| P
    P -->|"send_response"| S
    S -->|"return_response"| CR

    %% Build and Deployment Flow
    DS -->|"builds_container"| DC
    DC --> N

    %% Local Testing Simulation
    TEST -->|"simulate_request"| N

    %% External Integration
    CR -->|"external_call"| EXT
    EXT -->|"integrates_with"| N

    %% Styles
    classDef experimental fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef deployment fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef external fill:#fff3cd,stroke:#856404,stroke-width:2px;

    %% Click Events
    click KN1 "https://github.com/ourresearch/openalex-keywords/blob/main/v1/run_keywords.ipynb"
    click KN2 "https://github.com/ourresearch/openalex-keywords/blob/main/v1/store_keywords_SPARK.ipynb"
    click DC "https://github.com/ourresearch/openalex-keywords/tree/main/v2/003_Deployment/model_to_api/container"
    click DS "https://github.com/ourresearch/openalex-keywords/tree/main/v2/003_Deployment/model_to_api/container/Dockerfile"
    click DS "https://github.com/ourresearch/openalex-keywords/blob/main/v2/003_Deployment/model_to_api/container/build_and_push.sh"
    click N "https://github.com/ourresearch/openalex-keywords/tree/main/v2/003_Deployment/model_to_api/container/keyword_matcher"
    click TEST "https://github.com/ourresearch/openalex-keywords/tree/main/v2/003_Deployment/model_to_api/local_test"
    click EXT "https://github.com/ourresearch/openalex-keywords/tree/main/v2/003_Deployment/model_to_api/openalex-keyword-matcher-v2-endpoint"
```
