# boudinfl/pke architecture

> Architecture diagram of the GitHub repository boudinfl/pke, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/boudinfl/pke
- Repository: https://github.com/boudinfl/pke

## Overview

##### Project type and purpose
- This is an open-source Python toolkit (“pke”) for keyphrase extraction.
- It exposes a standardized API and CLI for loading documents, selecting candidate keyphrases, weighting them, and returning the top N.
- It ships with multiple unsupervised (graph-based, statistical) and one supervised model, and provides example scripts, tests, and Sphinx documentation.

##### File structure
- Top-level:
- pke/             ← core library package
- examples/        ← usage scripts and notebooks demonstrating functionality
- docs/            ← Sphinx sources and built HTML
- tests/           ← pytest test suite and sample data
- setup.py, requirements.txt ← packaging and dependencies
- .github/, .travis.yml ← CI workflows
- In pke/:
- base.py         ← abstract extractor class defining the pipeline
- readers.py      ← document loading and parsing
- data_structures.py, utils.py ← shared in-memory representations and helpers
- supervised/     ← feature-based supervised models (Kea, Wingnus)
- unsupervised/   ← multiple subpackages for statistical and graph-based models

##### Key points from the README
- Pip-installable package; depends on spaCy models
- Minimal 5-step extraction API (load_document → candidate_selection → candidate_weighting → get_n_best)
- Multiple extraction algorithms implemented as interchangeable classes

##### Components
a. Core components:
1. API/CLI Layer
- Entrypoint classes in base.py
- Example scripts in examples/
2. Document Reader
- pke.readers: handles raw text, spacy tokenization & POS tagging
3. Candidate Selector
- Implemented per-model in supervised/feature_based and unsupervised/*
4. Weighting Engine
- Graph algorithms (PageRank variants, multipartite rank, etc.)
- Statistical scorers (TfIdf, YAKE, etc.)
5. Ranking & Output
- get_n_best: final sorting and thresholding
6. Data Structures & Utilities
- Graph representations, frequency maps, POS filters
7. External Dependencies
- spaCy (NLP preprocessing)
- numpy, networkx or scipy (under the hood of some models)
8. Extension Points
- New models plugging into the base API
b. Relationships & data flows:
- Reader → spaCy preprocessing
- Preprocessed doc → Candidate Selector → Data Structures
- Data Structures → Weighting Engine
- Weighting Engine → Ranking & Output
- Output back to API/CLI (console print, return list, file write)
- Tests/Examples import API layer to validate/invoke pipelines
- CI/CD pipelines run tests and build docs automatically
c. Architectural patterns & principles:
- Strategy pattern: interchangeable extractor subclasses for each model
- Pipeline pattern: sequential steps (load, select, weight, rank)
- Modular separation: core API vs. supervised vs. unsupervised implementations
- Extensibility: add a new model by subclassing base.Extractor and implementing selection/weighting
d. Technologies & frameworks:
- Python packaging (setup.py, MANIFEST.in)
- spaCy for tokenization & POS
- Sphinx for docs under docs/source
- pytest for tests under tests/
- GitHub Actions & Travis CI for continuous integration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% API/CLI Layer
    subgraph "API/CLI Layer"
        API["base.Extractor\n(pke/base.py)"]:::module
        CLI1["keyphrase-extraction.py\n(examples/)"]:::module
        CLI2["benchmarking-models.py\n(examples/)"]:::module
    end

    %% Document Reader
    Reader["Document Reader\n(pke/readers.py)"]:::module

    %% External Dependency for NLP
    SpaCy["spaCy\n(Tokenization & POS)"]:::external

    %% Candidate Selector
    subgraph "Candidate Selector"
        direction TB
        subgraph "Supervised Models"
            KB["kea.py"]:::module
            WG["wingnus.py"]:::module
        end
        subgraph "Unsupervised Statistical"
            FP["firstphrases.py"]:::module
            KM["kpminer.py"]:::module
            TF["tfidf.py"]:::module
            YA["yake.py"]:::module
        end
        subgraph "Unsupervised Graph-Based"
            TR["textrank.py"]:::module
            SR["singlerank.py"]:::module
            TPR["single_tpr.py"]:::module
            PR["positionrank.py"]:::module
            MR["multipartiterank.py"]:::module
            TP["topicrank.py"]:::module
        end
    end

    %% Weighting & Ranking
    Weight["Weighting Engine\n(graph & statistical)"]:::module
    Rank["Ranking & Output\n(get_n_best)"]:::module

    %% Data Structures & Utilities
    DS["Data Structures\n(pke/data_structures.py)"]:::module
    Utils["Utilities\n(pke/utils.py)"]:::module

    %% Extension Points
    Ext["Extension Point\n(new Extractor subclass)"]:::module
    API --> Ext

    %% Examples & Tests
    Examples["Examples Directory\n(examples/)"]:::aux
    Tests["Tests Directory\n(tests/)"]:::aux

    %% Documentation & CI
    subgraph "Docs & CI"
        DocsSrc["Sphinx Sources\n(docs/source/)"]:::aux
        DocsBuild["HTML Build\n(docs/build/html/)"]:::aux
        CI1["GitHub Actions\n(.github/workflows/python-package.yml)"]:::aux
        CI2["Travis CI\n(.travis.yml)"]:::aux
    end

    %% Data Flow
    API -->|"load_document"| Reader
    Reader -->|"tokenize & POS"| SpaCy
    SpaCy -->|"preprocessed doc"| KB
    SpaCy --> SR
    SpaCy --> FP
    KB --> DS
    WG --> DS
    FP --> DS
    KM --> DS
    TF --> DS
    YA --> DS
    TR --> DS
    SR --> DS
    TPR --> DS
    PR --> DS
    MR --> DS
    TP --> DS
    DS --> UtilDS[Utilities] 
    UtilDS --> Weight
    Weight --> Rank
    Rank -->|"output keyphrases"| API

    %% Usage & Verification
    Examples --> API
    Tests --> API
    CI1 --> Tests
    CI1 --> DocsSrc
    CI2 --> Tests
    CI2 --> DocsSrc

    %% Click Events
    click API "https://github.com/boudinfl/pke/blob/master/pke/base.py"
    click CLI1 "https://github.com/boudinfl/pke/blob/master/examples/keyphrase-extraction.py"
    click CLI2 "https://github.com/boudinfl/pke/blob/master/examples/benchmarking-models.py"
    click Reader "https://github.com/boudinfl/pke/blob/master/pke/readers.py"
    click KB "https://github.com/boudinfl/pke/blob/master/pke/supervised/feature_based/kea.py"
    click WG "https://github.com/boudinfl/pke/blob/master/pke/supervised/feature_based/wingnus.py"
    click FP "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/statistical/firstphrases.py"
    click KM "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/statistical/kpminer.py"
    click TF "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/statistical/tfidf.py"
    click YA "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/statistical/yake.py"
    click TR "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/graph_based/textrank.py"
    click SR "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/graph_based/singlerank.py"
    click TPR "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/graph_based/single_tpr.py"
    click PR "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/graph_based/positionrank.py"
    click MR "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/graph_based/multipartiterank.py"
    click TP "https://github.com/boudinfl/pke/blob/master/pke/unsupervised/graph_based/topicrank.py"
    click DS "https://github.com/boudinfl/pke/blob/master/pke/data_structures.py"
    click UtilDS "https://github.com/boudinfl/pke/blob/master/pke/utils.py"
    click Ext "https://github.com/boudinfl/pke/blob/master/pke/supervised/api.py"
    click Examples "https://github.com/boudinfl/pke/tree/master/examples/"
    click Tests "https://github.com/boudinfl/pke/tree/master/tests/"
    click DocsSrc "https://github.com/boudinfl/pke/tree/master/docs/source/"
    click DocsBuild "https://github.com/boudinfl/pke/tree/master/docs/build/html/"
    click CI1 "https://github.com/boudinfl/pke/blob/master/.github/workflows/python-package.yml"
    click CI2 "https://github.com/boudinfl/pke/blob/master/.travis.yml"

    %% Styles
    classDef module fill:#cce5ff,stroke:#0366d6,stroke-width:1px;
    classDef external fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef aux fill:#fff3cd,stroke:#856404,stroke-width:1px;
```
