How egoist/sitefetch works
Key Components to Include:
CLI Interface Layer
• Entry point through src/cli.ts
• Command-line argument parsing
• User interaction components
Core Processing Layer
• Main functionality in src/index.ts (fetchSite)
• Markdown conversion (src/to-markdown.ts)
• URL processing and content fetching
Support Systems
• Logging system (src/logger.ts)
• Utility functions (src/utils.ts)
• Type definitions (src/types.ts)
Main Components (Top to Bottom)
User Input → CLI Layer → Core Processing → Output File
Detailed Flow (Include these interactions)
a. Input Processing
• CLI arguments (URL, concurrency, selectors)
• Configuration validation
• Pattern matching for URLs
b. Core Processing
• Web page fetching
• Concurrent processing management
• Readability extraction (mozilla/readability)
• Markdown conversion
• Content filtering based on selectors
c. Output Handling
• File system operations
• Logging and feedback
Use these visual elements:
• Blue: Core processing components
• Green: Input/Output operations
• Yellow: Utility functions
• Gray: External dependencies
• Double lines: Concurrent operations
Key Relationships to Highlight:
URL Processing Flow
Input URL → Pattern Matching → Concurrent Fetching → Content Extraction
Content Processing Flow
Raw HTML → Readability Processing → Markdown Conversion → Text Output
Configuration Flow
CLI Arguments → Validation → Processing Options → Execution
Special Considerations:
Concurrency Handling
• Parallel processing of multiple URLs
• Concurrency limits
External Dependencies
• mozilla/readability integration
• micromatch pattern matching
Error Handling
• Error paths
• Logging feedback loops
Level of Detail:
1. URL processing and matching logic
2. Content extraction and conversion pipeline
3. Concurrent execution management