How anduriroshan/carer-med-scraper works
Main Components
• Entry Point & Controller: The main entry is likely “app.py”, which (most probably) orchestrates the scraping operations, initializes modules, and possibly handles scheduling or routing.
• Crawling Modules:
• Article Crawlers: Contained in the “crawl_article” directory, with various Python scripts (e.g., crawl_article_allergy_sql.py, crawl_article_cardiology_sql.py, etc.) that handle scraping article data for each medical specialty.
• Page Crawlers: Found in the “crawl_page” directory, with a similar structure but designed for scraping page-based information.
• Utility Components: There are several “utils.py” files (in the top-level, crawl_article, and crawl_page directories) that likely provide common functionality to process and abstract the underlying scraping logic.
• Data/Processing Components: Files like “merging_collections.py”, “update.py”, and “query_search.py” indicate components that handle post-scraping data operations such as merging scraped data collections, updating existing data, and facilitating search queries.
• Auxiliary/Advanced Functionality: The “rag.py” file suggests there might be an integration with advanced processing such as a retrieval-augmented generation approach for content or search enhancement.
• Constants: “constanst.py” is likely used to store configuration or constant values used across the application.
System Interactions & Data Flow
• The flow likely begins at the main controller (app.py), which may determine what specialty or type of content to scrape.
• The controller routes the request to the appropriate crawling module:
• Article scraping modules (from the crawl_article directory) will be invoked when the system needs to scrape article-based content for a specific medical specialty.
• Similarly, page scraping modules (from the crawl_page directory) are called for page-based contents.
• Each of these modules uses common utility functions to handle tasks like HTTP requests, parsing, and error handling.
• After crawling, the data likely flows into a processing pipeline where merging (merging_collections.py) and updates (update.py) are performed. The query_search.py module might interface with either an internal index or an external search service.
• The advanced “rag.py” module, if integrated, might process the scraped data to enhance search relevance or content summarization.
Architectural Patterns and Design Principles
• Separation of Concerns: The clear split into article versus page crawler directories indicates a modular design where each component concerns itself with a distinct functionality.
• Reusability: The presence of multiple “utils.py” files in different directories suggests that common functionality is abstracted for reuse across various modules.
• Scalability: With each medical specialty having its own module, it’s straightforward to add new specialties or modify existing ones without disrupting the rest of the system.
• Maintainability: Use of a main entry point (app.py) to orchestrate actions and constants defined in “constanst.py” helps centralize configuration and keep the code maintainable.