duyennh2401/web-crawler-seg301

This is a multi-source focused news crawler. An operator runs main.py, which creates a shared BFS engine and independently processes five newspaper adapters. The engine enforces robots.txt, fetches pages, parses and filters links, persists pages and link relationships in SQLite, and returns article records. main.py combines those records into a digest stored in SQLite. The repository also provides a database inspection utility. Unsampled files such as digest.py and config.py are represented only where README or sampled callers establish their roles.