How chbhuvan/medical-report-summarization works
Project Type and Purpose
• The project is a specialized medical report summarization system.
• It leverages multiple NLP models (e.g., BERT and Pegasus) for generating summaries from clinical texts and includes a Named-Entity Recognition (NER) component.
• The ancillary files (e.g., PowerPoint presentation, PDF final report, Excel score sheets) indicate both development documentation and performance evaluation aspects.
Main Components
a. Data Sources and Ingestion:
• A directory (“beth_train”) holds numerous clinical report text files. This forms the primary input dataset.
• Adding an “Input Data” component where raw clinical reports are stored.
b. Preprocessing and Information Extraction:
• A preprocessing stage should be represented, where clinical text is possibly cleaned and prepared.
• The presence of NER.ipynb and extract_information2.py highlights an information extraction module that likely identifies key clinical entities.
c. Summarization Modules:
• There are two distinct summarization pipelines:
• BERT-based Summarization: Represented by the BERT_summary directory and associated script(s) like bioBERT2.py. This pipeline processes inputs and generates multiple summary files (e.g., summary_0.txt … summary_97.txt).
• Pegasus-based Summarization: Shown by the Pegasus_summary directory and possibly orchestrated via Pegasus.py. Similar to the BERT branch, it outputs a range of summaries.
d. NER Component:
• A component depicting the NER workflow. Input text (or possibly even the summaries) gets processed to extract more granular clinical data (indicated by files in the NER_output directory).
e. Evaluation and Reporting:
• The presence of summary score Excel files (summary_Scores.xlsx and summary_scores_pegasus.xlsx) suggests that there is an evaluation component that compares performance between models.
• A reporting/documentation block linked to the summary output (e.g., CS6120_NLP_Final_Report.pdf, Medical Report Summarization.pptx).
f. Auxiliary Buttons/Modules:
• Mention a caching mechanism (like __pycache__) if relevant, though not a primary architectural concern.
• You may optionally include “Utility Scripts” that support operations such as information extraction (extract_information2.py) and any configuration routines.
Relationships and Interactions
• Once processed, route the cleaned text to both the BERT-based and Pegasus-based summarization modules.
• Each summarizer outputs its text files stored in their respective directories.
• The outputs can then feed into the NER module to further extract and annotate clinical entities.
• Performance evaluation components take in generated summaries and provide scoring and analysis, which then feed into the documentation/reporting outputs.
Architectural Patterns and Technologies
• A pipeline or service-oriented architecture where each key stage (data preprocessing, summarization, NER, evaluation) is clearly divided.
• Design principles include modularity and separation of concerns, so each summarization engine and the NER component are isolated but interact with standardized input/output.
• Use of deep learning frameworks (presumably TensorFlow or PyTorch, given the models) and Python-based scripting.