How neulab/reviewadvisor works
Project type and purpose
• This is an open-source NLP research toolkit (“ReviewAdvisor”) that automates scientific peer review by applying an “extract-then-generate” pipeline. It consists of two main processing modules—an extractor and a tagger—plus supporting scripts for data retrieval, preprocessing, and post-processing.
File structure
• Top-level:
• download_dataset.sh & download_tagger.sh – data/model acquisition scripts
• extractor/ – section-based and statistical extraction implementation
• tagger/ – sequence labeling (“tagging”) pipeline with helper utilities, config, and scripts
• scripts/ – generic download helper (gdown.pl)
• fig/ & materials/ – documentation, figures, annotation guidelines, sample JSON
• Orchestration point: the shell scripts call Python modules in extractor/ and tagger/ in sequence.
Essential components
a. Data Acquisition
• download_dataset.sh → populates dataset/; download_tagger.sh → retrieves pretrained model into tagger/seqlab_final/
b. Extraction Module (extractor/)
• extractor.py, extractor_utils.py read raw papers & parameters.txt → produce extracted snippets or selection indices
• example.ipynb demonstrates usage interactively
c. Tagging Module (tagger/)
• run_tagger.py or tasks.py orchestrates:
• prepare.sh → splits data (split.py), jsonlize outputs (jsonlize.py)
• annotator.py & heuristics.py → rule–based tagging for certain labels
• utils_batch.py & post_process.sh → batch inference using pretrained seqlab_final model
• config.json and labels.txt govern model behavior
d. (Optional) Generation / Evaluation / Visualization
• Evaluation and bias-analysis figures (in fig/) can be represented as “Metrics & Visualization” component.
Relationships & data flow
• Scripts (download_*.sh, prepare.sh, post_process.sh) surround and trigger each major module
Architectural patterns & design principles
• Modular pipeline: clear separation of concerns (data download, extraction logic, tagging logic, post-processing)
• Script-driven orchestration (shell scripts invoking Python modules)
• Configuration-driven components (config.json, parameters.txt) for flexibility
• Helper utilities extracted into a “helper/” package within tagger
Technologies & frameworks
• Python + standard libraries (os, json, argparse)
• Shell scripting (bash) for orchestration
• Jupyter notebook for interactive demos (extractor/example.ipynb)
• External pretrained sequence labeling model (loaded via tagger/run_tagger.py)
Detail to capture
• Shell scripts as the “glue” that invoke each stage
• The contents of tagger/helper (split, heuristics, jsonlize) as sub-components of the Tagging Module
• The role of config.json and labels.txt in controlling tagging behavior
• Interactive/demo components (notebook, fig/) as auxiliary outputs