How leduonng/crawl-truyenfull works
And Illustrate the Main Components
• User Input: Label a component showing how users invoke the crawl either via the command line (python main.py with URL) or via run.bat (which prompts for the URL).
• Crawler Orchestrator: This is the central component in main.py that coordinates the crawling process.
• Chapter Discovery Module: Represent a subcomponent that scans the pagination to determine the total number of chapters. Note in the README that it looks for the “Cuối” link or the largest chapter number.
• Content Parser/Cleaner: Include a module that applies BeautifulSoup to clean the HTML, removes ads, and ensures only the required content is extracted.
• Storage Manager: This module handles file writing:
• A File Saver for the complete story as a Word document (using python-docx).
• A File Splitter to produce segmented text files (each ~18000 words).
• Logging Mechanism: A component dedicated to logging each step of the process into the “logs” directory, capturing errors and progress.
Relationships and Data Flow
• The Orchestrator then delegates the job to the Chapter Discovery module to identify the chapter range.
• Once chapters are identified, the Multi-threaded Crawling Engine concurrently fetches content by interacting with the Content Parser module.
• Each fetched chapter's content flows to the Storage Manager, which saves the data into the corresponding file formats.
• Parallelly, all major steps and errors are reported to the Logging Mechanism.
Key Architectural Patterns and Design Principles
• Modularity: Each of the above components is logically separated and responsible for a specific task, which makes the system easier to manage and extend.
• Resilience: Incorporate error handling and retry logic. Illustrate how the system continues despite encountering empty chapters or errors.
• Batch Processing: Batch handling (crawling 10 chapters per batch) is an important internal detail that can be noted.
Relevant Technologies and Frameworks
• Language: Python 3.6+
• HTTP Requests: The “requests” library for fetching web content.
• HTML Parsing: “beautifulsoup4” for processing and cleaning page content.
• Document Generation: “python-docx” for creating Word documents.
• Operating Environment: Windows is indirectly referenced by run.bat.