# abhadimishra/scrap_bs4_textual_analysis architecture

> Architecture diagram of the GitHub repository abhadimishra/scrap_bs4_textual_analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhadimishra/scrap_bs4_textual_analysis
- Repository: https://github.com/abhadimishra/scrap_bs4_textual_analysis

## Overview

#### Purpose and execution model
This repository is a Jupyter-based batch pipeline for scraping article text from URL records, preprocessing it, computing sentiment/readability metrics, and writing results into an Excel output template. The primary entry point is `main.ipynb`; there is no application server or package structure.

Paths: `main.ipynb`, `README.md`, `Output Data Structure.xlsx`.

#### Input/output contract
The workflow expects an `Input.xlsx` workbook (referenced by the notebook README, not present in the tree) containing URL and ID columns. IDs associate scraped text files with source rows. The provided `Output Data Structure.xlsx` is the result schema/template updated with calculated fields.

Paths: `main.ipynb`, `Output Data Structure.xlsx`.

#### Stage 1: web extraction
A `data_extraction` function uses HTTP requests and BeautifulSoup to retrieve each input URL and extract page text. Extracted content is persisted as individual text files, creating a disk boundary between network collection and later NLP processing.

External dependencies/services: arbitrary target websites over HTTP, `requests`, `BeautifulSoup`.

Path: `main.ipynb`.

#### Stage 2: stopword preprocessing
The notebook loads stopword lists from files into a set, iterates the extracted-text directory, and rewrites text files after filtering stopwords. This is a destructive transformation: subsequent scoring and metrics consume the filtered file contents rather than raw page text.

Runtime/data dependencies: Python, filesystem-hosted stopword lists, NLTK-related text processing.

Path: `main.ipynb`.

#### Stage 3: sentiment scoring
For each processed text file, the pipeline compares tokens against positive and negative word lists, producing positive score, negative score, polarity score, and subjectivity score. Results are matched back to the corresponding input ID and written into output workbook columns.

Paths: `main.ipynb`, `Output Data Structure.xlsx`.

#### Stage 4: textual-complexity analysis
The same processed corpus is analyzed for sentence and word-level measures: average sentence length, complex-word percentage/count, syllable count per word, average words per sentence, personal-pronoun count, and average word length. These calculations extend the row-level output record alongside sentiment values.

Paths: `main.ipynb`, `Output Data Structure.xlsx`.

#### Persistence and operational boundaries
Excel is both the input/output integration boundary and the final deliverable; text files are the intermediate corpus boundary. The notebook relies on local working-directory conventions for input workbooks, lexicons/stopword files, and extracted text directories. It requires Python with `pandas`, `requests`, BeautifulSoup, `os`, and `nltk`, plus network access during extraction.

Paths: `main.ipynb`, `Output Data Structure.xlsx`.

Main entry points: [`main.ipynb`](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb) (Metrics by ID), [`README.md`](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/README.md) (README.md), [`Output Data Structure.xlsx`](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/Output%20Data%20Structure.xlsx) (Output Data Structure.xlsx)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs"]
  node_input_workbook["Input.xlsx<br/>URL and ID workbook"]
  node_target_websites{{"Target websites<br/>HTTP sources"}}
  node_stopword_files["Stopword lists<br/>local lexicons"]
  node_sentiment_lexicons["Positive/negative lists<br/>local lexicons"]
  node_readme["README.md<br/>workflow documentation<br/>[README.md]"]
end

subgraph group_pipeline["Notebook Pipeline"]
  node_main_notebook["main.ipynb<br/>Jupyter entry point<br/>[main.ipynb]"]
  node_extractor["Web extraction<br/>requests + BeautifulSoup<br/>[main.ipynb]"]
  node_raw_corpus[("Extracted text files<br/>intermediate corpus")]
  node_preprocessor["Stopword preprocessing<br/>NLTK text processing<br/>[main.ipynb]"]
  node_processed_corpus[("Processed text files<br/>scoring corpus")]
  node_sentiment["Sentiment scoring<br/>token lexicon analysis<br/>[main.ipynb]"]
  node_complexity["Text complexity analysis<br/>readability metrics<br/>[main.ipynb]"]
  node_row_results["Metrics by ID<br/>row-level records<br/>[main.ipynb]"]
end

subgraph group_outputs["Output"]
  node_output_template["Output Data Structure.xlsx<br/>Excel result template"]
  node_final_workbook["Completed output workbook<br/>Excel deliverable"]
end

node_readme -.->|"documents"| node_input_workbook
node_input_workbook -->|"URLs and IDs"| node_extractor
node_target_websites -->|"HTTP responses"| node_extractor
node_main_notebook -->|"executes"| node_extractor
node_extractor -->|"persists text"| node_raw_corpus
node_stopword_files -->|"filter terms"| node_preprocessor
node_main_notebook -->|"executes"| node_preprocessor
node_raw_corpus -->|"rewrites"| node_preprocessor
node_preprocessor -->|"filtered text"| node_processed_corpus
node_sentiment_lexicons -->|"word matches"| node_sentiment
node_processed_corpus -->|"tokens"| node_sentiment
node_processed_corpus -->|"text measures"| node_complexity
node_main_notebook -->|"executes"| node_sentiment
node_main_notebook -->|"executes"| node_complexity
node_sentiment -->|"sentiment fields"| node_row_results
node_complexity -->|"complexity fields"| node_row_results
node_input_workbook -->|"ID association"| node_row_results
node_output_template -->|"column schema"| node_row_results
node_row_results -->|"writes values"| node_final_workbook

click node_main_notebook "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb"
click node_extractor "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb"
click node_preprocessor "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb"
click node_sentiment "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb"
click node_complexity "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb"
click node_row_results "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb"
click node_output_template "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/Output%20Data%20Structure.xlsx"
click node_final_workbook "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/Output%20Data%20Structure.xlsx"
click node_readme "https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_input_workbook,node_target_websites,node_stopword_files,node_sentiment_lexicons,node_readme toneBlue
class node_main_notebook,node_extractor,node_raw_corpus,node_preprocessor,node_processed_corpus,node_sentiment,node_complexity,node_row_results toneAmber
class node_output_template,node_final_workbook toneMint
```

## Components

### Inputs

- **Input.xlsx**: URL and ID workbook · Expected local input; not included in the repository.
- **Target websites**: HTTP sources
- **Stopword lists**: local lexicons
- **Positive/negative lists**: local lexicons
- [README.md](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/README.md): workflow documentation · Documents expected input workbook conventions. · `README.md`

### Notebook Pipeline

- [main.ipynb](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb): Jupyter entry point · Runs the end-to-end batch workflow. · `main.ipynb`
- [Web extraction](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb): requests + BeautifulSoup · Retrieves URLs and extracts page text. · `main.ipynb`
- **Extracted text files**: intermediate corpus · One text file per source record.
- [Stopword preprocessing](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb): NLTK text processing · Destructively rewrites corpus files after filtering stopwords. · `main.ipynb`
- **Processed text files**: scoring corpus · Filtered files consumed by all analyses.
- [Sentiment scoring](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb): token lexicon analysis · Computes positive, negative, polarity, and subjectivity scores. · `main.ipynb`
- [Text complexity analysis](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb): readability metrics · Computes sentence, word, syllable, and pronoun measures. · `main.ipynb`
- [Metrics by ID](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/main.ipynb): row-level records · Joins analysis results to input workbook rows. · `main.ipynb`

### Output

- [Output Data Structure.xlsx](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/Output%20Data%20Structure.xlsx): Excel result template · Defines and receives calculated output columns. · `Output Data Structure.xlsx`
- [Completed output workbook](https://github.com/abhadimishra/scrap_bs4_textual_analysis/blob/main/Output%20Data%20Structure.xlsx): Excel deliverable · `Output Data Structure.xlsx`

## Connections

- README.md → Input.xlsx: documents
- Input.xlsx → Web extraction: URLs and IDs
- Target websites → Web extraction: HTTP responses
- main.ipynb → Web extraction: executes
- Web extraction → Extracted text files: persists text
- Stopword lists → Stopword preprocessing: filter terms
- main.ipynb → Stopword preprocessing: executes
- Extracted text files → Stopword preprocessing: rewrites
- Stopword preprocessing → Processed text files: filtered text
- Positive/negative lists → Sentiment scoring: word matches
- Processed text files → Sentiment scoring: tokens
- Processed text files → Text complexity analysis: text measures
- main.ipynb → Sentiment scoring: executes
- main.ipynb → Text complexity analysis: executes
- Sentiment scoring → Metrics by ID: sentiment fields
- Text complexity analysis → Metrics by ID: complexity fields
- Input.xlsx → Metrics by ID: ID association
- Output Data Structure.xlsx → Metrics by ID: column schema
- Metrics by ID → Completed output workbook: writes values