# cxcscmu/crawl4llm architecture

> Architecture diagram of the GitHub repository cxcscmu/crawl4llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cxcscmu/crawl4llm
- Repository: https://github.com/cxcscmu/crawl4llm

## Overview

##### Project Type and Purpose
- This is a specialized crawler tool used to simulate web crawling and selectively score documents (using methods like fastText-based scoring, length, random, or inlink count) from a large dataset.
- It’s not a typical full-stack application; rather, it’s a command-line tool built with a modular design that supports different crawling and scoring methods for controlled document preselection.

##### Analysis of the File Structure
- Top-Level Scripts: Files like crawl.py, crawler.py, and fetch_docs.py suggest the main orchestration of the crawl process as well as later stages of document fetching.
- Rating and Scoring: document_rater.py, normalizer.py, and the scoring configuration embedded in the YAML files (create under configs/) clearly separate the concerns of ranking documents using different techniques.
- Data Access: access_data.py and corpus_interface.py provide utilities for interfacing with the ClueWeb22 dataset, while seed.txt provides the seed list of initial documents.
- Logging & Utility: wandb_logger.py emphasizes integration with external tools (Weights & Biases) for experiment tracking; utils.py likely contains shared helper functions.
- Notice the absence of a directory structure—everything is file-oriented in the root—so components are split at the script level rather than by folder, yet the README still introduces a conceptual separation (e.g., configurations in configs/).

A. Identify Main Components:
- Configuration Component:
▪ External YAML configuration files (not in file tree but referenced) that set up paths (cw22_root_path, output_dir), selected scoring methods, and other parameters.
- Crawling Orchestrator:
▪ crawl.py: the entry point that reads configuration, initializes the crawler, and coordinates its execution.
▪ crawler.py: encapsulates crawling logic, processing of document queues, managing worker threads (or processes) for parallel work.
- Document Rating/Scoring Module:
▪ document_rater.py and normalizer.py: implement the rating methods (length, fasttext_score, random_score, and inlink_count) to rank the documents.
- Document Access/Fetching Module:
▪ fetch_docs.py: post-crawl, retrieves the actual document texts using document ids.
▪ access_data.py: for on-demand inspection of specific documents from the ClueWeb22 dataset.
- Data Interface:
▪ corpus_interface.py: likely provides interfaces to read and interact with the ClueWeb22 dataset.
- Logging & Experiment Tracking:
▪ wandb_logger.py: integrates with the external logging service (Weights & Biases) to track metrics and run metadata.
- Utility Functions:
▪ utils.py: supports common functionalities used across modules.
- Seed Definition:
▪ seed.txt: provides initial seed document IDs to start the crawl process.

B. Identify Relationships and Interactions:
- The process begins with reading an external YAML configuration that defines runtime parameters.
- crawl.py uses these configurations to initialize and control the crawling process contained in crawler.py.
- During crawling, the document rating module (document_rater.py and associated scorers defined by the YAML configuration) evaluates and ranks documents.
- The crawler relies on corpus_interface.py to interact with the underlying dataset (ClueWeb22) and to fetch outlinks or metadata.
- Logging is done concurrently via wandb_logger.py, which ties back to the external wandb service.
- Once crawl records (document IDs) are stored (in paths defined by the configuration, e.g., output_dir), fetch_docs.py is used to retrieve full text documents for the next LLM pretraining phase.
- Optionally, access_data.py provides a way to inspect individual documents.

C. Highlight Architectural Patterns / Design Principles:
- Modularity: Each file is focused on a single aspect of the process (crawling, fetching, scoring, logging, etc.).
- Configuration Driven: Use of YAML configuration files allows for flexibility and separation of environment-specific parameters.
- Parallel Processing: The configuration mentions “num_workers”, indicating concurrency is integral to achieve efficient crawling.
- Extensibility: The “rating_methods” abstraction allows for plugging in additional document-scoring strategies without modifying the core logic.

D. Relevant Technologies / Frameworks:
- Python (>= 3.10) is the primary language.
- Third-party libraries including numpy, tqdm for progress indication, fastText for model-based scoring, pyyaml for configuration parsing, and wandb for logging.
- External datasets and models: ClueWeb22 for the data and fastText classifiers downloaded from Hugging Face.

##### Elements
▪ Configuration (YAML files)
▪ Crawl Orchestrator (crawl.py, crawler.py)
▪ Document Rating Modules (document_rater.py, normalizer.py)
▪ Data Access Layer (corpus_interface.py, fetch_docs.py, access_data.py)
▪ Logging and Monitoring (wandb_logger.py with wandb integration)
▪ Utility and Seed Data (utils.py, seed.txt)
▪ Data flow from configuration to crawling engine.
▪ Interaction between the crawler and the scoring module.
▪ Data flow from the crawling process into the output directory and subsequently into the document fetching module.
▪ Integration paths linking the crawler and logging systems.
▪ Command-line tools (scripts that are executed directly).
▪ Modules that serve as services (for data interaction, logging, rating).
▪ External dependencies (datasets, fastText models, and wandb).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Components
    config["Configuration (YAML)"]:::config
    seed["seed.txt"]:::seed
    utils["utils.py"]:::utility

    %% Crawl Orchestrator Subgraph
    subgraph "Crawl Orchestrator"
        crawlpy["crawl.py"]:::orchestrator
        crawlerpy["crawler.py"]:::orchestrator
    end

    %% Document Rating/Scoring Module Subgraph
    subgraph "Document Rating/Scoring Module"
        documentrater["document_rater.py"]:::rating
        normalizer["normalizer.py"]:::rating
    end

    %% Data Interface
    corpusinterface["corpus_interface.py"]:::dataInterface

    %% Logging & Experiment Tracking
    wandblogger["wandb_logger.py"]:::logging

    %% Document Access/Fetching Module Subgraph
    subgraph "Document Access/Fetching Module"
        fetchdocs["fetch_docs.py"]:::access
        accessdata["access_data.py"]:::access
    end

    %% Connections
    config -->|"feeds"| crawlpy
    seed -->|"initial seeds"| crawlpy
    crawlpy -->|"orchestrates"| crawlerpy
    utils -->|"support functions"| crawlerpy

    crawlerpy -->|"uses"| corpusinterface
    crawlerpy -->|"invokes"| wandblogger
    crawlerpy -->|"triggers"| documentrater
    corpusinterface -->|"supplies data"| documentrater

    documentrater -->|"informs fetching"| fetchdocs
    crawlerpy -->|"outputs IDs to"| fetchdocs

    %% Click Events
    click crawlpy "https://github.com/cxcscmu/crawl4llm/blob/main/crawl.py"
    click crawlerpy "https://github.com/cxcscmu/crawl4llm/blob/main/crawler.py"
    click documentrater "https://github.com/cxcscmu/crawl4llm/blob/main/document_rater.py"
    click normalizer "https://github.com/cxcscmu/crawl4llm/blob/main/normalizer.py"
    click fetchdocs "https://github.com/cxcscmu/crawl4llm/blob/main/fetch_docs.py"
    click accessdata "https://github.com/cxcscmu/crawl4llm/blob/main/access_data.py"
    click corpusinterface "https://github.com/cxcscmu/crawl4llm/blob/main/corpus_interface.py"
    click wandblogger "https://github.com/cxcscmu/crawl4llm/blob/main/wandb_logger.py"
    click utils "https://github.com/cxcscmu/crawl4llm/blob/main/utils.py"
    click seed "https://github.com/cxcscmu/crawl4llm/blob/main/seed.txt"

    %% Styles
    classDef config fill:#FAD02E,stroke:#333,stroke-width:2px;
    classDef orchestrator fill:#B0E0E6,stroke:#333,stroke-width:2px;
    classDef rating fill:#C1FFC1,stroke:#333,stroke-width:2px;
    classDef access fill:#FFE4E1,stroke:#333,stroke-width:2px;
    classDef dataInterface fill:#D3D3D3,stroke:#333,stroke-width:2px;
    classDef logging fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef utility fill:#E6E6FA,stroke:#333,stroke-width:2px;
    classDef seed fill:#FFB6C1,stroke:#333,stroke-width:2px;
```
