# egoist/sitefetch architecture

> Architecture diagram of the GitHub repository egoist/sitefetch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/egoist/sitefetch
- Repository: https://github.com/egoist/sitefetch

## Overview

Key Components to Include:

##### CLI Interface Layer
- Entry point through `src/cli.ts`
- Command-line argument parsing
- User interaction components

##### Core Processing Layer
- Main functionality in `src/index.ts` (fetchSite)
- Markdown conversion (`src/to-markdown.ts`)
- URL processing and content fetching

##### Support Systems
- Logging system (`src/logger.ts`)
- Utility functions (`src/utils.ts`)
- Type definitions (`src/types.ts`)

##### Main Components (Top to Bottom)

**User Input → CLI Layer → Core Processing → Output File**

##### Detailed Flow (Include these interactions)

a. Input Processing
- CLI arguments (URL, concurrency, selectors)
- Configuration validation
- Pattern matching for URLs

b. Core Processing
- Web page fetching
- Concurrent processing management
- Readability extraction (mozilla/readability)
- Markdown conversion
- Content filtering based on selectors

c. Output Handling
- File system operations
- Logging and feedback

Use these visual elements:

- Blue: Core processing components
- Green: Input/Output operations
- Yellow: Utility functions
- Gray: External dependencies

- Double lines: Concurrent operations

Key Relationships to Highlight:

##### URL Processing Flow
**Input URL → Pattern Matching → Concurrent Fetching → Content Extraction**

##### Content Processing Flow
**Raw HTML → Readability Processing → Markdown Conversion → Text Output**

##### Configuration Flow
**CLI Arguments → Validation → Processing Options → Execution**

Special Considerations:

##### Concurrency Handling
- Parallel processing of multiple URLs
- Concurrency limits

##### External Dependencies
- mozilla/readability integration
- micromatch pattern matching

##### Error Handling
- Error paths
- Logging feedback loops

Level of Detail:

1. URL processing and matching logic
2. Content extraction and conversion pipeline
3. Concurrent execution management

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Legend
        L1[Process]
        L2([Input/Output])
        L3{{Decision}}
        L4[(File System)]
        L5[/External Dependency/]
    end

    subgraph "CLI Interface"
        CLI[CLI Interface]:::core
        Args([Command Line Arguments]):::io
        Args --> CLI
        CLI --> ConfigValidation{Config Valid?}:::decision
    end

    subgraph "Core Processing"
        MainProcess[Main Processing]:::core
        URLProcess[URL Processing]:::core
        Concurrent[Concurrent Fetcher]:::core
        Convert[Content Converter]:::core
        
        MainProcess --> URLProcess
        URLProcess --> Concurrent
        Concurrent --> Convert
    end

    subgraph "Support Systems"
        Logger[Logger]:::utility
        Utils[Utilities]:::utility
        Types[Type Definitions]:::utility
    end

    subgraph "External Components"
        Readability[/mozilla-readability/]:::external
        Micromatch[/micromatch/]:::external
    end

    subgraph "Output Handling"
        FileOps[(File System Operations)]:::io
        OutputFile([Text Output]):::io
    end

    ConfigValidation -->|Yes| MainProcess
    ConfigValidation -->|No| Logger
    
    Convert --> FileOps
    FileOps --> OutputFile
    
    MainProcess --> Logger
    URLProcess --> Utils
    Concurrent --> Readability
    URLProcess --> Micromatch

    %% Component Mappings
    click CLI "https://github.com/egoist/sitefetch/blob/main/src/cli.ts"
    click MainProcess "https://github.com/egoist/sitefetch/blob/main/src/index.ts"
    click Convert "https://github.com/egoist/sitefetch/blob/main/src/to-markdown.ts"
    click Logger "https://github.com/egoist/sitefetch/blob/main/src/logger.ts"
    click Utils "https://github.com/egoist/sitefetch/blob/main/src/utils.ts"
    click Types "https://github.com/egoist/sitefetch/blob/main/src/types.ts"

    %% Styling
    classDef core fill:#2374ab,stroke:#2374ab,color:white
    classDef io fill:#57a773,stroke:#57a773,color:white
    classDef decision fill:#ffc857,stroke:#ffc857,color:black
    classDef utility fill:#a8a8a8,stroke:#a8a8a8,color:white
    classDef external fill:#df2935,stroke:#df2935,color:white
```
