How sahilnegi4444/higs_gat_based_summarization works
1. This repository is an offline research/training pipeline for multi-document abstractive summarization of Indian English news. The main boundary is between experiment code under scripts/ and model implementations under models/, with configuration-driven runs from configs/.
2. The central system is HiGS, a dual-encoder fusion summarizer implemented in models/higs_model.py. Its data path is: multiple related articles in → sentence splitting and tokenization → BERT sentence embeddings for structural modeling → adjacency matrix construction from similarity/entity overlap → GAT layers → projection to BART’s latent size → fusion with BART’s native encoder output → BART decoder generates the summary.
3. HiGS has an explicit structural vs. semantic split. Pathway A captures cross-document relations with BERT + graph attention; Pathway B preserves fluent generation with a frozen BART encoder/decoder backbone. The architecture relies on dual tokenization because BERT and BART do not share vocabularies or embedding spaces.
4. Training is staged and intentionally decoupled. scripts/train_higs.py runs phase 1 to train the graph-side components while keeping BART frozen, then phase 2 to unfreeze the decoder and align generation to the learned graph representations. This two-phase boundary is important to preserve pretrained language ability while adding factual grounding.
5. Baseline models are handled separately from HiGS. models/baselines.py and scripts/train_baselines.py cover encoder-decoder baselines such as PRIMERA, LED, LongT5, and Flan-T5-XL, while models/llm_lora.py covers large-model fine-tuning via LoRA/QLoRA for models like Mistral, Llama 3, Qwen2, and Gemma. The repo therefore compares from-scratch summarizers against parameter-efficient LLM adaptation.
6. Data preparation is centralized in scripts/prepare_data.py. The README indicates a clean → split → stats flow: strip HTML and deduplicate raw data, build train/validation/test splits, then compute dataset statistics and EDA plots. The cleaned dataset is expected externally in data/newssumm_cleaned.parquet, so raw data is a separate input boundary from the codebase.
7. Evaluation is split into two layers. scripts/evaluate_higs.py measures generation quality and runtime characteristics such as ROUGE, BERTScore, latency, throughput, and peak VRAM. scripts/run_local_consistency_check.py adds factuality checks using entity overlap, a DeBERTa-based NLI pass, and topic overlap, making factual consistency a distinct subsystem from lexical quality.
8. Inference/qualitative generation is isolated in scripts/generate_mds.py. It is meant to synthesize a summary from multiple articles directly, separate from training and batch evaluation. This gives the repo a clear “demo/inference” entry point apart from experiment execution.
9. The configs/ directory is the experiment control plane. Each YAML file maps to a specific model family or training regime, including HiGS and the baseline suites. Architecturally, this means model choice, training hyperparameters, and PEFT settings are externalized rather than hard-coded.
10. results/ is a persisted artifact layer, not just logs. It stores benchmark JSONs, evaluation reports, factual consistency outputs, generated samples, and a summary CSV. For diagramming, treat this as the output sink for all training/evaluation workflows rather than an active runtime component.
11. Supporting assets are split across docs/ and notebooks/. docs/architecture.md and the diagram image provide the formal mathematical/visual specification, while notebooks/dataset_analysis.ipynb is exploratory analysis rather than part of the production pipeline.
12. The runtime assumptions matter for the architecture: Python with PyTorch, GPU acceleration, CUDA, and spaCy for sentence splitting/named entity extraction. Those libraries primarily support the graph construction and factual consistency subsystems, especially entity-aware adjacency building and consistency analysis.
Main entry points: train_higs.py Train HiGS · train_baselines.py Train baselines · higs_model.py HiGS model · evaluate_higs.py Evaluate HiGS
Components 18
Control plane
Experiment configuration and run orchestration
- HiGS config · Configuration for the dual-encoder HiGS training regime.
configs/higs.yaml - Baseline configs · Configuration files for classical encoder-decoder baselines and PEFT LLM runs.
Model layer
Summarization model implementations
- HiGS model · Graph-grounded abstractive summarizer combining BERT, GAT, and BART.
models/higs_model.py - Baselines · Implementation of PRIMERA, LED, LongT5, and Flan-T5 style summarizers.
models/baselines.py - LLM LoRA · Parameter-efficient fine-tuning for large models using LoRA/QLoRA.
models/llm_lora.py
Pipeline scripts
Training, evaluation, data preparation, and inference entry points
- Prepare data · Cleans raw news, builds splits, and computes dataset statistics.
scripts/prepare_data.py - Train HiGS · Runs staged HiGS training with frozen then unfrozen BART components.
scripts/train_higs.py - Train baselines · Trains the encoder-decoder baseline suite from configuration.
scripts/train_baselines.py - Evaluate HiGS · Measures ROUGE, BERTScore, latency, throughput, and VRAM usage.
scripts/evaluate_higs.py - Consistency check · Runs entity overlap, NLI, and topic overlap checks for factual consistency.
scripts/run_local_consistency_check.py - Generate summary · Direct multi-document summary generation entry point.
scripts/generate_mds.py
Artifacts
Persisted outputs and reports
- Benchmarks · Stored benchmark JSON reports for model comparisons.
results/benchmark_reports - Eval reports · Stored evaluation outputs for model runs.
results/eval_reports - Factual report · Persisted factual consistency analysis output.
results/factual_consistency_report.json - Samples · Persisted generated summaries and example outputs.
results/generated_samples.json - Summary table · Aggregated benchmark comparison CSV.
results/benchmark_table.csv
Reference
Documentation and exploratory analysis
- Architecture doc · Formal description of the HiGS design and math.
docs/architecture.md - Data notebook · Exploratory dataset analysis notebook.
notebooks/dataset_analysis.ipynb
Connections 21
- HiGS config → Train HiGS: drives · HiGS configuration selects training regime and hyperparameters.
- Baseline configs → Train baselines: drives · Baseline and PEFT configuration files control experiment runs.
- Baseline configs → LLM LoRA: selects · Configuration options map to PEFT LLM fine-tuning setups.
- Prepare data → Train HiGS: feeds · Prepared splits provide training and validation inputs for HiGS.
- Prepare data → Train baselines: feeds · Prepared splits are reused by baseline training runs.
- HiGS model → Train HiGS: implements · HiGS model is the core architecture trained in the staged pipeline.
- Baselines → Train baselines: implements · Baseline model implementations are exercised by the baseline trainer.
- LLM LoRA → Train baselines: extends · PEFT LLM adaptation is part of the comparison training suite.
- Train HiGS → Evaluate HiGS: evaluated by · Trained HiGS checkpoints are scored by the evaluation script.
- Train HiGS → Generate summary: served by · HiGS checkpoints can be used for qualitative summary generation.
- Train baselines → Benchmarks: writes · Baseline runs emit benchmark JSON artifacts.
- Evaluate HiGS → Benchmarks: writes · Generation evaluation produces benchmark reports.
- Evaluate HiGS → Eval reports: writes · Per-run evaluation outputs are stored as JSON reports.
- Consistency check → Factual report: writes · Factual consistency analysis is persisted as a dedicated report.
- Generate summary → Samples: writes · Inference produces generated summary samples for inspection.
- Benchmarks → Summary table: aggregates · Benchmark reports are summarized into a comparison table.
- Architecture doc → HiGS model: documents · Documentation describes the HiGS architecture and data flow.
- Data notebook → Prepare data: analyzes · Notebook supports exploratory analysis of the dataset preparation workflow.
- HiGS model → Evaluate HiGS: measured by · HiGS outputs are assessed on quality and runtime metrics.
- HiGS model → Consistency check: checked by · HiGS generations can be inspected for factual consistency.
- Generate summary → Consistency check: reviewed by · Generated summaries can be passed to factual consistency checks.