How haoshuai999/news-summarization works
Main Components
a. Data Ingestion & Preprocessing:
• Dataset ingestion of the CNN/Daily Mail corpus.
• Data cleaning, tokenization, and any filtering steps executed in the "Final_project_preprocessing.ipynb".
b. Embedding Generation:
• That embeddings are then fed into the model.
c. Model Architecture (Bidirectional LSTM):
d. Evaluation & Outputs:
• An evaluation component where metrics such as Rough-1 score (0.351) are computed.
• Production of summaries that are extracted from the articles.
Relationships and Interactions
• From raw news articles input → Preprocessing module.
• Preprocessing module → Embedding Generation component.
• Embeddings → BiLSTM summarization model.
• BiLSTM model → Evaluation & output of summarized content.
Architectural Patterns & Design Principles
• The project follows a modular structure. Each of the notebooks represents a clearly separated phase of the processing pipeline.
• The pipeline can be seen as a chain of data transformations where each module has a single responsibility: input preparation, feature extraction (embeddings), and modeling.
• Data flow orchestration, illustrating how each pipeline stage depends on the previous stage.
Technologies, Frameworks, or Libraries
• Mention the use of Jupyter Notebook for experimentation and documentation.
• Although not explicitly listed, the bidirectional LSTM suggests the use of deep learning libraries such as TensorFlow or PyTorch.
• Data preprocessing might involve libraries like pandas, NLTK, or spaCy.