How abhadimishra/scrap_bs4_textual_analysis works
Purpose and execution model
This repository is a Jupyter-based batch pipeline for scraping article text from URL records, preprocessing it, computing sentiment/readability metrics, and writing results into an Excel output template. The primary entry point is main.ipynb; there is no application server or package structure.
Paths: main.ipynb, README.md, Output Data Structure.xlsx.
Input/output contract
The workflow expects an Input.xlsx workbook (referenced by the notebook README, not present in the tree) containing URL and ID columns. IDs associate scraped text files with source rows. The provided Output Data Structure.xlsx is the result schema/template updated with calculated fields.
Paths: main.ipynb, Output Data Structure.xlsx.
Stage 1: web extraction
A data_extraction function uses HTTP requests and BeautifulSoup to retrieve each input URL and extract page text. Extracted content is persisted as individual text files, creating a disk boundary between network collection and later NLP processing.
External dependencies/services: arbitrary target websites over HTTP, requests, BeautifulSoup.
Path: main.ipynb.
Stage 2: stopword preprocessing
The notebook loads stopword lists from files into a set, iterates the extracted-text directory, and rewrites text files after filtering stopwords. This is a destructive transformation: subsequent scoring and metrics consume the filtered file contents rather than raw page text.
Runtime/data dependencies: Python, filesystem-hosted stopword lists, NLTK-related text processing.
Path: main.ipynb.
Stage 3: sentiment scoring
For each processed text file, the pipeline compares tokens against positive and negative word lists, producing positive score, negative score, polarity score, and subjectivity score. Results are matched back to the corresponding input ID and written into output workbook columns.
Paths: main.ipynb, Output Data Structure.xlsx.
Stage 4: textual-complexity analysis
The same processed corpus is analyzed for sentence and word-level measures: average sentence length, complex-word percentage/count, syllable count per word, average words per sentence, personal-pronoun count, and average word length. These calculations extend the row-level output record alongside sentiment values.
Paths: main.ipynb, Output Data Structure.xlsx.
Persistence and operational boundaries
Excel is both the input/output integration boundary and the final deliverable; text files are the intermediate corpus boundary. The notebook relies on local working-directory conventions for input workbooks, lexicons/stopword files, and extracted text directories. It requires Python with pandas, requests, BeautifulSoup, os, and nltk, plus network access during extraction.
Paths: main.ipynb, Output Data Structure.xlsx.
Main entry points: main.ipynb Metrics by ID · README.md README.md · Output Data Structure.xlsx Output Data Structure.xlsx
Components 15
Inputs
- Input.xlsx · Expected local input; not included in the repository.
- Target websites
- Stopword lists
- Positive/negative lists
- README.md · Documents expected input workbook conventions.
README.md
Notebook Pipeline
- main.ipynb · Runs the end-to-end batch workflow.
main.ipynb - Web extraction · Retrieves URLs and extracts page text.
main.ipynb - Extracted text files · One text file per source record.
- Stopword preprocessing · Destructively rewrites corpus files after filtering stopwords.
main.ipynb - Processed text files · Filtered files consumed by all analyses.
- Sentiment scoring · Computes positive, negative, polarity, and subjectivity scores.
main.ipynb - Text complexity analysis · Computes sentence, word, syllable, and pronoun measures.
main.ipynb - Metrics by ID · Joins analysis results to input workbook rows.
main.ipynb
Output
- Output Data Structure.xlsx · Defines and receives calculated output columns.
Output Data Structure.xlsx - Completed output workbook
Output Data Structure.xlsx
Connections 19
- README.md → Input.xlsx: documents
- Input.xlsx → Web extraction: URLs and IDs
- Target websites → Web extraction: HTTP responses
- main.ipynb → Web extraction: executes
- Web extraction → Extracted text files: persists text
- Stopword lists → Stopword preprocessing: filter terms
- main.ipynb → Stopword preprocessing: executes
- Extracted text files → Stopword preprocessing: rewrites
- Stopword preprocessing → Processed text files: filtered text
- Positive/negative lists → Sentiment scoring: word matches
- Processed text files → Sentiment scoring: tokens
- Processed text files → Text complexity analysis: text measures
- main.ipynb → Sentiment scoring: executes
- main.ipynb → Text complexity analysis: executes
- Sentiment scoring → Metrics by ID: sentiment fields
- Text complexity analysis → Metrics by ID: complexity fields
- Input.xlsx → Metrics by ID: ID association
- Output Data Structure.xlsx → Metrics by ID: column schema
- Metrics by ID → Completed output workbook: writes values