How boudinfl/pke works
Project type and purpose
• This is an open-source Python toolkit (“pke”) for keyphrase extraction.
• It exposes a standardized API and CLI for loading documents, selecting candidate keyphrases, weighting them, and returning the top N.
• It ships with multiple unsupervised (graph-based, statistical) and one supervised model, and provides example scripts, tests, and Sphinx documentation.
File structure
• Top-level:
• pke/ ← core library package
• examples/ ← usage scripts and notebooks demonstrating functionality
• docs/ ← Sphinx sources and built HTML
• tests/ ← pytest test suite and sample data
• setup.py, requirements.txt ← packaging and dependencies
• .github/, .travis.yml ← CI workflows
• In pke/:
• base.py ← abstract extractor class defining the pipeline
• readers.py ← document loading and parsing
• data_structures.py, utils.py ← shared in-memory representations and helpers
• supervised/ ← feature-based supervised models (Kea, Wingnus)
• unsupervised/ ← multiple subpackages for statistical and graph-based models
Key points from the README
• Pip-installable package; depends on spaCy models
• Minimal 5-step extraction API (load_document → candidate_selection → candidate_weighting → get_n_best)
• Multiple extraction algorithms implemented as interchangeable classes
Components
a. Core components:
1. API/CLI Layer
• Entrypoint classes in base.py
• Example scripts in examples/
2. Document Reader
• pke.readers: handles raw text, spacy tokenization & POS tagging
3. Candidate Selector
• Implemented per-model in supervised/feature_based and unsupervised/*
4. Weighting Engine
• Graph algorithms (PageRank variants, multipartite rank, etc.)
• Statistical scorers (TfIdf, YAKE, etc.)
5. Ranking & Output
• get_n_best: final sorting and thresholding
6. Data Structures & Utilities
• Graph representations, frequency maps, POS filters
7. External Dependencies
• spaCy (NLP preprocessing)
• numpy, networkx or scipy (under the hood of some models)
8. Extension Points
• New models plugging into the base API
b. Relationships & data flows:
• Reader → spaCy preprocessing
• Preprocessed doc → Candidate Selector → Data Structures
• Data Structures → Weighting Engine
• Weighting Engine → Ranking & Output
• Output back to API/CLI (console print, return list, file write)
• Tests/Examples import API layer to validate/invoke pipelines
• CI/CD pipelines run tests and build docs automatically
c. Architectural patterns & principles:
• Strategy pattern: interchangeable extractor subclasses for each model
• Pipeline pattern: sequential steps (load, select, weight, rank)
• Modular separation: core API vs. supervised vs. unsupervised implementations
• Extensibility: add a new model by subclassing base.Extractor and implementing selection/weighting
d. Technologies & frameworks:
• Python packaging (setup.py, MANIFEST.in)
• spaCy for tokenization & POS
• Sphinx for docs under docs/source
• pytest for tests under tests/
• GitHub Actions & Travis CI for continuous integration