# hucheng-beauty/study architecture

> Architecture diagram of the GitHub repository hucheng-beauty/study, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hucheng-beauty/study
- Repository: https://github.com/hucheng-beauty/study

## Overview

##### Main Domains/Components
- Command-Line Applications:
- External Sort (located under cmd/external_sort)
- Possibly executable entry points in internal/crawler and internal/crawler_distributed
- Web Crawler Functionality:
- Core crawler modules (internal/crawler) featuring engine (with both concurrent and simple engines), fetcher, model, persist, scheduler, and specialized parser (for “zhenai”).
- Distributed version under internal/crawler_distributed, which also includes configuration, persistence (both client and server sides), RPC support, and worker modules.
- Golang Learning and Experimentation Modules:
- Data Structures & Algorithms (internal/data_structure_and_algorithm): covering arrays, lists, queues, stacks, and exercises.
- Design Patterns (internal/design_pattern): including factory modes (abstract, factory method, static) and policy mode.
- Multiple Golang feature demos (internal/golang): examples in concurrency (atomic, channel patterns, mutex, semaphore), error handling (errgroup, recover), HTTP & Gin servers, RPC, web engine, and tests (using Gomock and standard testing patterns).
- Other Utility/Demo Components:
- Log Processing (internal/log_process), Pipeline (internal/pipeline), Sorting algorithms (internal/sort)
- Package tips (pkg/tips/string) demonstrating string utilities or best practices

- Application Layer / Executables (cmd folder)
- Crawler Module (with its subdivisions for simple and distributed versions)
- Algorithms & Data Structures Library
- Design Pattern Demonstration Module
- Golang Essentials & Concurrency/Networking Demos (from internal/golang)
- Additional utilities (log process, pipeline, sort) and helper packages (pkg)

b. For each primary domain, further detail the internal structure:
- For the crawler:
- Detail components like Engine (with branching for “concurrent.go”, “simple.go” and “worker.go”), Fetcher, Scheduler (queued vs. simple), and Persist.
- How the distributed crawler version uses separate RPC support, client/server implementations for persistence and workers.
- For the Golang demos:
- Various experiments according to domain (e.g., concurrency demos, HTTP/web server demos, error handling, and interface examples).
- Associate each folder with its main executable or demonstration code.

c. Interactions and relationships:
- The crawler feeding fetched data into the persist module or forwarding requests to the scheduler.
- Command-line apps (like external_sort) receiving input, processing data, and outputting results.
- Dependencies where one module imports functionalities from another (for example, parts of internal/golang might be used by or compared to similar implementations in other sections).

##### Architectural Patterns and Principles
- Separation of concerns: different parts of the system handle crawling (fetching, parsing, persisting) separately.
- For distributed components, expose client–server and RPC communication patterns that decouple the worker nodes.
- Within the design pattern demos, note the usage of common patterns (Factory, Policy) to illustrate modularity and extensibility.
- For the Golang examples, indicate how concurrency patterns (channels, mutexes, semaphores, context usage) are demonstrated.
- Design principles, such as separation between demo code and production-like code (even if it is a testbed), which modularizes the examples.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Command-Line Applications"
        A1["External Sort App"]:::cmd
        A2["Crawler Executable"]:::cmd
        A3["Distributed Crawler Executable"]:::cmd
    end

    subgraph "Web Crawler Functionality"
        B1["Crawler Engine (Concurrent/Simple)"]:::crawler
        B2["Fetcher Module"]:::crawler
        B3["Data Models"]:::crawler
        B4["Persistence Module"]:::crawler
        B5["Scheduler Module"]:::crawler
        B6["Specialized Zhenai Parsers"]:::crawler
    end

    subgraph "Distributed Crawler"
        C1["Configuration"]:::distributed
        C2["Distributed Persistence (Client/Server)"]:::distributed
        C3["RPC Support"]:::distributed
        C4["Worker Module"]:::distributed
    end

    subgraph "Data Structures & Algorithms Library"
        D1["DS & Algorithms"]:::dsa
    end

    subgraph "Design Pattern Demonstration"
        E1["Design Patterns"]:::design
    end

    subgraph "Golang Essentials & Concurrency/Networking Demos"
        F1["Golang Demos"]:::golang
    end

    subgraph "Utility & Demo Components"
        G1["Log Processing"]:::utility
        G2["Pipeline"]:::utility
        G3["Sorting Algorithms"]:::utility
    end

    subgraph "Helper Packages"
        H1["String Tips"]:::helper
    end

    %% Relationships for Command-Line Applications to Crawler
    A2 -->|"uses"| B1
    B2 -->|"feeds"| B1
    B1 -->|"storesTo"| B4
    B1 -->|"schedulesVia"| B5
    B5 -->|"parsesUsing"| B6
    B1 -->|"mapsTo"| B3

    %% Relationships for Distributed Crawler
    A3 -->|"configures"| C1
    A3 -->|"calls"| C3
    A3 -->|"handles"| C2
    A3 -->|"delegatesTo"| C4
    C3 -->|"communicatesWith"| C4

    %% Cross-Domain Interactions
    F1 -->|"references"| E1
    F1 -->|"comparesWith"| D1

    %% Click Events
    click A1 "https://github.com/hucheng-beauty/study/tree/master/cmd/external_sort"
    click A2 "https://github.com/hucheng-beauty/study/blob/master/internal/crawler/main.go"
    click A3 "https://github.com/hucheng-beauty/study/blob/master/internal/crawler_distributed/main.go"
    click B1 "https://github.com/hucheng-beauty/study/tree/master/internal/crawler/engine"
    click B2 "https://github.com/hucheng-beauty/study/blob/master/internal/crawler/fetcher/fetcher.go"
    click B3 "https://github.com/hucheng-beauty/study/tree/master/internal/crawler/model"
    click B4 "https://github.com/hucheng-beauty/study/blob/master/internal/crawler/persist/itemsaver.go"
    click B5 "https://github.com/hucheng-beauty/study/tree/master/internal/crawler/scheduler"
    click B6 "https://github.com/hucheng-beauty/study/tree/master/internal/crawler/zhenai/parser"
    click C1 "https://github.com/hucheng-beauty/study/blob/master/internal/crawler_distributed/config/config.go"
    click C2 "https://github.com/hucheng-beauty/study/tree/master/internal/crawler_distributed/persist"
    click C3 "https://github.com/hucheng-beauty/study/blob/master/internal/crawler_distributed/rpcsupport/rpc.go"
    click C4 "https://github.com/hucheng-beauty/study/tree/master/internal/crawler_distributed/worker"
    click D1 "https://github.com/hucheng-beauty/study/tree/master/internal/data_structure_and_algorithm"
    click E1 "https://github.com/hucheng-beauty/study/tree/master/internal/design_pattern"
    click F1 "https://github.com/hucheng-beauty/study/tree/master/internal/golang"
    click G1 "https://github.com/hucheng-beauty/study/blob/master/internal/log_process/main.go"
    click G2 "https://github.com/hucheng-beauty/study/tree/master/internal/pipeline"
    click G3 "https://github.com/hucheng-beauty/study/tree/master/internal/sort"
    click H1 "https://github.com/hucheng-beauty/study/tree/master/pkg/tips/string"

    %% Styles
    classDef cmd fill:#FFDDCC,stroke:#FF6600,stroke-width:2px;
    classDef crawler fill:#CCE5FF,stroke:#3366CC,stroke-width:2px;
    classDef distributed fill:#CCFFCC,stroke:#339933,stroke-width:2px;
    classDef dsa fill:#E6CCFF,stroke:#9933FF,stroke-width:2px;
    classDef design fill:#FFFFCC,stroke:#FFCC00,stroke-width:2px;
    classDef golang fill:#FFCCCC,stroke:#FF6666,stroke-width:2px;
    classDef utility fill:#FFE6CC,stroke:#FF9900,stroke-width:2px;
    classDef helper fill:#F0F0F0,stroke:#999999,stroke-width:2px;
```
