# mustakabsarkhan/dse_company_scraper_python architecture

> Architecture diagram of the GitHub repository mustakabsarkhan/dse_company_scraper_python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mustakabsarkhan/dse_company_scraper_python
- Repository: https://github.com/mustakabsarkhan/dse_company_scraper_python

## Overview

1. **What this repo is**
   - This is a Python 3.11, async data-pipeline architecture for collecting Dhaka Stock Exchange company fundamentals from messy public HTML and exporting a research workbook.
   - The public repo is an **architecture preview** only: the runnable scraping implementation is intentionally withheld, and `python main.py` is meant to exit with a preview-only message.

2. **End-to-end system shape**
   - The pipeline is staged as: discover sectors → discover companies → fetch company pages → parse and normalize fields → compute screening/quality metrics → export Excel.
   - The project is not a web app; the main boundary is between external DSE pages and internal normalized financial analysis.

3. **Entry point and orchestration**
   - `main.py` is the top-level orchestrator.
   - It coordinates the pipeline modules rather than holding scraping logic itself, which keeps the execution flow centralized and the subsystems separated.

4. **Discovery pipeline**
   - `pipelines/sectors.py` is the first stage: it discovers tradable sector links and filters out non-equity-like groups.
   - `pipelines/companies.py` expands those sectors into company detail URLs.
   - This creates the crawl frontier before any detailed company fetches happen.

5. **Company-level data collection**
   - `pipelines/company_info.py` represents the company profile stage boundary.
   - It is responsible for taking a company URL and driving the retrieval/parsing path that produces the company’s structured fundamentals.

6. **Core networking and execution concerns**
   - `core/client.py` is the async HTTP/client boundary, including throttling and retry behavior in the private implementation.
   - The README describes bounded concurrency, dynamic pacing after failures, and jittered retries, so the network layer is designed to protect the source site and tolerate transient errors.

7. **Parsing and normalization boundary**
   - `core/parser.py` is the key transformation layer: it turns inconsistent page structures into a normalized schema.
   - The repo description emphasizes layered extraction, defensive null handling, duplicate prevention, and stable output ordering so the downstream analysis can compare companies and runs reliably.

8. **Market calendar and logging support**
   - `core/holidays.py` provides the market-calendar boundary used to reason about trading days and scheduled runs.
   - `core/logger.py` is the shared logging boundary across the pipeline, important for traceability in a long-running scrape/export job.

9. **Data quality and analysis output**
   - The private runtime calculates screening metrics, sector comparisons, and data-quality flags after normalization.
   - The analysis layer intentionally separates usable metrics from unreliable or missing ones, with sector medians, trimmed averages, outlier flags, and “value trap”/“speculative” style labels for screening.

10. **Excel export subsystem**
   - `export/excel.py` is the workbook-generation boundary.
   - The output is a multi-sheet Excel file under `Export_Data/DSE_Data_{Market_Date}.xlsx`, with sheets for guide text, raw cleaned data, processed analysis, sector summary, watchlist, quality issues, and dropped empty columns.

11. **Configuration and environment**
   - `config.py` is the central configuration boundary for the runtime.
   - `requirements.txt` defines the Python dependency set used by the private implementation, but the public repo does not expose the production scraper details.

12. **Automation and scheduling**
   - The architecture includes scheduled/manual execution in production, with runs intended after the Bangladesh market close on Sunday through Thursday.
   - `docs/automation-architecture.yml` documents that workflow, but there is no active `.github/workflows/` automation in this public repo.

13. **External dependency boundary**
   - The only material external system is the Dhaka Stock Exchange website, which acts as the source of unstructured company and sector data.
   - The architecture’s main purpose is to convert that unstable source into a repeatable, research-friendly internal dataset and workbook.

Main entry points: [`main.py`](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/main.py) (Main), [`pipelines/company_info.py`](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/company_info.py) (Company Info), [`export/excel.py`](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/export/excel.py) (Excel Export), [`pipelines/sectors.py`](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/sectors.py) (Sectors)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_main["Main<br/>entrypoint<br/>[main.py]"]
  node_config["Config<br/>[config.py]"]
end

subgraph group_discovery["Discovery"]
  node_sectors["Sectors<br/>discovery stage<br/>[sectors.py]"]
  node_companies["Companies<br/>crawl frontier<br/>[companies.py]"]
  node_company_info["Company Info<br/>profile stage<br/>[company_info.py]"]
end

subgraph group_core["Core"]
  node_client["HTTP Client<br/>async transport<br/>[client.py]"]
  node_parser["Parser<br/>normalizer<br/>[parser.py]"]
  node_logger["Logger<br/>logging<br/>[logger.py]"]
  node_holidays["Holidays<br/>market calendar<br/>[holidays.py]"]
end

subgraph group_output["Output"]
  node_export_excel["Excel Export<br/>workbook writer<br/>[excel.py]"]
  node_analysis{{"Analysis<br/>screening metrics"}}
  node_watchlist["Watchlist<br/>research slice"]
  node_quality["Quality Issues<br/>data quality"]
end

subgraph group_docs["Docs"]
  node_automation["Automation<br/>runbook"]
end

subgraph group_external["External Source"]
  node_dse_site(("DSE Site<br/>external source"))
end

node_main -->|"start"| node_sectors
node_main -->|"continue"| node_companies
node_main -->|"collect"| node_company_info
node_main -->|"finish"| node_export_excel
node_main -->|"reads"| node_config
node_sectors -->|"fetch"| node_dse_site
node_companies -->|"fetch"| node_dse_site
node_company_info -->|"fetch"| node_dse_site
node_company_info -->|"uses"| node_client
node_company_info -->|"normalizes"| node_parser
node_client -->|"logs"| node_logger
node_parser -->|"logs"| node_logger
node_main -->|"schedules"| node_holidays
node_analysis -->|"feeds"| node_export_excel
node_quality -->|"feeds"| node_export_excel
node_watchlist -->|"feeds"| node_export_excel
node_automation -.->|"describes"| node_main

click node_main "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/main.py"
click node_config "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/config.py"
click node_sectors "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/sectors.py"
click node_companies "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/companies.py"
click node_company_info "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/company_info.py"
click node_client "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/client.py"
click node_parser "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/parser.py"
click node_logger "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/logger.py"
click node_holidays "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/holidays.py"
click node_export_excel "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/export/excel.py"
click node_automation "https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/docs/automation-architecture.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config toneBlue
class node_sectors,node_companies,node_company_info toneAmber
class node_client,node_parser,node_logger,node_holidays toneMint
class node_export_excel,node_analysis,node_watchlist,node_quality toneRose
class node_automation toneIndigo
class node_dse_site toneTeal
```

## Components

### Orchestration

Top-level entry and runtime coordination

- [Main](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/main.py): entrypoint · Preview-only orchestrator that coordinates the pipeline · `main.py`
- [Config](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/config.py): runtime config · Central configuration for execution and environment · `config.py`

### Discovery

Sector and company frontier building

- [Sectors](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/sectors.py): discovery stage · Finds tradable sector links and filters non-equity groups · `pipelines/sectors.py`
- [Companies](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/companies.py): crawl frontier · Expands sectors into company detail URLs · `pipelines/companies.py`
- [Company Info](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/pipelines/company_info.py): profile stage · Fetches and drives structured company fundamentals extraction · `pipelines/company_info.py`

### Core

Networking, parsing, calendar, and logging boundaries

- [HTTP Client](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/client.py): async transport · Async request boundary with throttling and retry behavior · `core/client.py`
- [Parser](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/parser.py): normalizer · Turns messy pages into a stable normalized schema · `core/parser.py`
- [Logger](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/logger.py): logging · Shared logging boundary for pipeline traceability · `core/logger.py`
- [Holidays](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/core/holidays.py): market calendar · Trading-day and scheduled-run calendar logic · `core/holidays.py`

### Output

Analysis and workbook generation

- [Excel Export](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/export/excel.py): workbook writer · Builds the multi-sheet research workbook · `export/excel.py`
- **Analysis**: screening metrics · Computes quality flags, medians, and sector comparisons
- **Watchlist**: research slice · Flags names of interest after screening
- **Quality Issues**: data quality · Separates missing, unreliable, and dropped fields

### Docs

Operational notes and automation design

- [Automation](https://github.com/mustakabsarkhan/dse_company_scraper_python/blob/main/docs/automation-architecture.yml): runbook · Scheduled/manual execution design for market-close runs · `docs/automation-architecture.yml`

### External Source

Public DSE pages and unstructured HTML

- **DSE Site**: external source · Public Dhaka Stock Exchange pages supplying unstable HTML

## Connections

- Main → Sectors: start · Begins sector discovery stage
- Main → Companies: continue · Coordinates crawl frontier expansion
- Main → Company Info: collect · Hands company URLs into the profile stage
- Main → Excel Export: finish · Triggers workbook generation after processing
- Main → Config: reads · Loads runtime settings and environment values
- Sectors → DSE Site: fetch · Discovers sector pages from the public site
- Companies → DSE Site: fetch · Expands sector listings into company URLs
- Company Info → DSE Site: fetch · Retrieves individual company pages
- Company Info → HTTP Client: uses · Relies on async HTTP transport and retry policy
- Company Info → Parser: normalizes · Converts inconsistent HTML into structured fields
- HTTP Client → Logger: logs · Records request pacing, retries, and failures
- Parser → Logger: logs · Records extraction and normalization issues
- Main → Holidays: schedules · Uses trading calendar to reason about run timing
- Analysis → Excel Export: feeds · Supplies processed metrics to the workbook writer
- Quality Issues → Excel Export: feeds · Provides issues and dropped-column reporting
- Watchlist → Excel Export: feeds · Supplies curated screening output
- Automation → Main: describes · Documents intended scheduled and manual execution flow