mustakabsarkhan/dse_company_scraper_python

1. What this repo is

- This is a Python 3.11, async data-pipeline architecture for collecting Dhaka Stock Exchange company fundamentals from messy public HTML and exporting a research workbook.

- The public repo is an architecture preview only: the runnable scraping implementation is intentionally withheld, and python main.py is meant to exit with a preview-only message.

2. End-to-end system shape

- The pipeline is staged as: discover sectors → discover companies → fetch company pages → parse and normalize fields → compute screening/quality metrics → export Excel.

- The project is not a web app; the main boundary is between external DSE pages and internal normalized financial analysis.

3. Entry point and orchestration

- main.py is the top-level orchestrator.

- It coordinates the pipeline modules rather than holding scraping logic itself, which keeps the execution flow centralized and the subsystems separated.

4. Discovery pipeline

- pipelines/sectors.py is the first stage: it discovers tradable sector links and filters out non-equity-like groups.

- pipelines/companies.py expands those sectors into company detail URLs.

- This creates the crawl frontier before any detailed company fetches happen.

5. Company-level data collection

- pipelines/company_info.py represents the company profile stage boundary.

- It is responsible for taking a company URL and driving the retrieval/parsing path that produces the company’s structured fundamentals.

6. Core networking and execution concerns

- core/client.py is the async HTTP/client boundary, including throttling and retry behavior in the private implementation.

- The README describes bounded concurrency, dynamic pacing after failures, and jittered retries, so the network layer is designed to protect the source site and tolerate transient errors.

7. Parsing and normalization boundary

- core/parser.py is the key transformation layer: it turns inconsistent page structures into a normalized schema.

- The repo description emphasizes layered extraction, defensive null handling, duplicate prevention, and stable output ordering so the downstream analysis can compare companies and runs reliably.

8. Market calendar and logging support

- core/holidays.py provides the market-calendar boundary used to reason about trading days and scheduled runs.

- core/logger.py is the shared logging boundary across the pipeline, important for traceability in a long-running scrape/export job.

9. Data quality and analysis output

- The private runtime calculates screening metrics, sector comparisons, and data-quality flags after normalization.

- The analysis layer intentionally separates usable metrics from unreliable or missing ones, with sector medians, trimmed averages, outlier flags, and “value trap”/“speculative” style labels for screening.

10. Excel export subsystem

- export/excel.py is the workbook-generation boundary.

- The output is a multi-sheet Excel file under Export_Data/DSE_Data_{Market_Date}.xlsx, with sheets for guide text, raw cleaned data, processed analysis, sector summary, watchlist, quality issues, and dropped empty columns.

11. Configuration and environment

- config.py is the central configuration boundary for the runtime.

- requirements.txt defines the Python dependency set used by the private implementation, but the public repo does not expose the production scraper details.

12. Automation and scheduling

- The architecture includes scheduled/manual execution in production, with runs intended after the Bangladesh market close on Sunday through Thursday.

- docs/automation-architecture.yml documents that workflow, but there is no active .github/workflows/ automation in this public repo.

13. External dependency boundary

- The only material external system is the Dhaka Stock Exchange website, which acts as the source of unstructured company and sector data.

- The architecture’s main purpose is to convert that unstable source into a repeatable, research-friendly internal dataset and workbook.