How yanlinf/casent works
Project type and purpose
• CASENT is an open-source NLP library and command-line toolkit for ultra-fine entity typing and extraction.
• It wraps a seq2seq T5 backbone with calibration, label‐space management (UFET and WikiData ontologies), dataset utilities, and optional span detection.
File structure
• Top‐level “casent/” directory contains the core Python package:
• app.py – REST/CLI entrypoint for serving predictions
• entity_typing_t5.py – Predictor classes (API and CLI wrappers)
• calibration.py / dataset.py / trie.py – Model calibration and dataset handling
• utils/ – Helpers for LLM backends, NER utils, SPARQL queries, visualization
• “scripts/” holds orchestration scripts: data download, benchmarking, evaluation, serving.
• “ontology_data/” holds static label vocabularies and mappings (JSON, CSV).
• Training/inference/evaluation drivers at top level: train_t5.py, predict_t5.py, evaluate_ufet_prediction.py.
README for insights
• Core components: T5 seq2seq model, UFET and WikiData ontologies, span detector (optional).
• Usage modes: raw API calls, CLI scripts, demonstration server.
• External dependencies: HuggingFace model hub, SPARQL endpoint for ontology lookups, stanza parser.
Main components and their interactions
a. Core Library (“casent/” Python package)
• Model Module: wraps T5ForConditionalGeneration with calibration layer
• Dataset Module: loads UFET/WikiData labels, builds prefix‐trie for decoding
• Utils Module:
– llm_backend: abstracts HF/LLM calls
– ner_utils & ufet_utils: preprocessing and postprocessing
– wikidata_sparql & ontology loaders: external ontology services
– visualization: output formatting
b. CLI & API Layer
• train_t5.py: orchestrates data loading → model instantiation → training loop
• predict_t5.py: loads checkpoint → runs batched inference → dumps JSON
• evaluate_ufet_prediction.py: computes metrics on predictions
• scripts/serve_casent_app.sh & casent/app.py: spins up REST service
c. Data & Ontology Layer
• ontology_data/: static JSON/CSV used at runtime for label mapping
• scripts/download_ufet.sh: fetches raw UFET data
d. External Services
• HuggingFace Hub: pre-trained T5 checkpoints
• Wikidata SPARQL endpoint: optional RDF lookups
Relationships:
– Training pipeline pulls raw data → dataset module → core model → checkpoint
– Inference pipeline loads checkpoint → predictor → utils postprocess → output
– Serving pipeline wraps inference in REST endpoints
Architectural patterns & design principles
• Layered design:
• Data Layer (ontology_data, scripts)
• Core Logic Layer (casent package)
• Interface Layer (CLI scripts, REST via app.py)
• Single‐pass autoregressive decoding with calibrated confidence
• Modular utilities for ontology and parser extensibility