# esurharun/cado-nfs architecture

> Architecture diagram of the GitHub repository esurharun/cado-nfs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/esurharun/cado-nfs
- Repository: https://github.com/esurharun/cado-nfs

## Overview

##### Project type and purpose
- Cado-NFS is an open-source, command-line driven toolset implementing the Number Field Sieve algorithm to factor large integers.
- It comprises two major “flavors” of interface:
a. A monolithic CLI (`cado-nfs.py`) that orchestrates all steps locally or via SSH.
b. A client/server mode (under `scripts/cadofactor/`) where a Python-based server dispatches work to independent clients.
- Under the hood, a collection of C/C++ binaries cover each stage: polynomial selection, sieving, filtering, linear algebra, square root, plus support libraries (GF2X, linear algebra, utilities).

##### Key directories and their roles
- `/` (root) – orchestration scripts, top-level Makefile/CMake, entry-point `cado-nfs.py`, `cado-nfs-client.py`.
- `config/`, `.gitlab-ci.yml`, `ci/` – build-system checks, platform-detection tests, CI pipelines.
- `gf2x/` – stand-alone library for GF(2^n) arithmetic.
- `linalg/` – linear algebra routines (including parallel MPI/OpenMP implementations).
- `polyselect/` – polynomial selection code and Sage bindings.
- `sieve/` – lattice sieving code, ECM & bucket sieving.
- `filter/` – relation filtering and logging stages.
- `sqrt/` – square-root CRT and specialized routines.
- `utils/` + `misc/` – shared utilities (memory managers, multi-precision support, thread pools, JSON, etc.).
- `scripts/` – Python scripts:
- `cadofactor/` – server, client, database backends (dictdb, MySQL), REST API (`api_server.py`).
- `descent/` – algebraic descent helpers.
- build, lint, packaging tools, Docker wrappers.
- `parameters/` – default parameter sets for various factorization scenarios.
- `tests/` – unit and integration tests for each component.

##### System components
A. User Interface
- `cado-nfs.py` CLI
- `scripts/cadofactor/api_server.py` (Flask-based REST server)
- `cado-nfs-client.py` CLI for client mode
B. Orchestration Layer (Python)
- Parameter parser → decides on local vs. distributed execution
- Task launcher → spawns processes or SSH commands
- In server mode, REST endpoints manage workunit lifecycle
C. Data Storage
- In-memory/dictdb (default) or persistent storage via SQLite/MySQL
- Parameter files and snapshots under `parameters/`
D. Compute Layer (binaries)
1. polyselect (C/C++ + Sage glue)
2. sieve (C++ sieving engines + ECM)
3. filter (C code for relation filtering)
4. linalg (C++ MPI/OpenMP linear algebra modules)
5. sqrt (C code for square root step)
E. Utility Libraries
- GF2X library
- utils (multi-precision, threading, memory, JSON, etc.)
F. External Services & Tools
- SSH for remote client launch
- Docker (optional containers)
- CI pipelines (.gitlab-ci.yml)
- Third-party libs: GMP, MPIR, hwloc, MPI, OpenMP

##### Interactions and data flow
- User runs `cado-nfs.py` → orchestration layer reads parameter files →
- Local mode: directly invokes compute binaries in sequence or in parallel.
- SSH mode: uses SSH to launch `cado-nfs-client.py` on remote hosts.
- Server mode: Python REST server enqueues tasks in DB → clients poll REST API → fetch workunits → run binaries → post results back.
- Each compute stage reads/writes intermediate files under the work directory.
- Filtering and linear algebra may exchange data via files or MPI.

##### Architectural patterns & principles
- Modular pipeline: each algorithmic step is encapsulated in its own binary.
- Layered orchestration: Python script sits atop the C/C++ compute layer.
- Adapter pattern: multiple DB backends, multiple scheduling modes (local, SSH, REST).
- Plugin/extensibility points: parameter files, strategy scripts under `sieve/strategies`.
- Build-time configurability via CMake and Make abstractions.

##### Essential details to capture
- Separation of concerns: orchestration vs. compute
- Multiple execution modes (local, SSH, REST)
- Data persistence (parameter files, database)
- Use of parallelism (threads, MPI)
- Extensibility (strategies, parameter sets)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    subgraph "User Interface"
        direction TB
        CLI_Mono["CLI Orchestrator\n(cado-nfs.py)"]:::blue
        CLI_Client["CLI Client\n(cado-nfs-client.py)"]:::blue
        REST_Server["REST Server\n(api_server.py)"]:::blue
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        ParamParser["Parameter Parser"]:::blue
        TaskLauncher["Task Launcher"]:::blue
        REST_Endpoints["REST Endpoints"]:::blue

        subgraph "Core Python Logic"
            direction TB
            TopLevel["toplevel.py"]:::blue
            CadoTask["cadotask.py"]:::blue
            CadoCommand["cadocommand.py"]:::blue
            CadoParams["cadoparams.py"]:::blue
            CadoPrograms["cadoprograms.py"]:::blue
        end

        subgraph "Database Backends"
            direction TB
            DictDB["dictdb.py"]:::blue
            SQLite3["sqlite3.py"]:::blue
            MySQL["mysql.py"]:::blue
            DBFactory["factory.py"]:::blue
        end
    end

    %% Data Storage
    Parameter_Files["Parameter Files\n(parameters/)"]:::gray
    Persistent_DB["Persistent DB\n(SQLite/MySQL)"]:::gray

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        PolySelect["Polynomial Selection\n(polyselect/)"]:::green
        Sieve["Sieving Engines\n(sieve/)"]:::green
        Filter["Relation Filtering\n(filter/)"]:::green
        LinAlg["Linear Algebra\n(linalg/)"]:::green
        Sqrt["Square-Root Step\n(sqrt/)"]:::green
    end

    %% Utility Libraries
    subgraph "Utility Libraries"
        direction TB
        GF2X["GF(2^n) Arithmetic\n(gf2x/)"]:::orange
        Utils["Shared Utilities\n(utils/)"]:::orange
    end

    %% External Services & Tools
    SSH["SSH"]:::gray
    Docker["Docker"]:::gray
    CI["CI Pipelines\n(.gitlab-ci.yml)"]:::gray

    %% Connections
    CLI_Mono -->|reads| Parameter_Files
    CLI_Mono -->|invokes| TaskLauncher
    CLI_Mono -->|invokes| ParamParser
    CLI_Mono -->|local/parallel| PolySelect
    CLI_Mono -->|local/parallel| Sieve
    CLI_Mono -->|local/parallel| Filter
    CLI_Mono -->|local/parallel| LinAlg
    CLI_Mono -->|local/parallel| Sqrt
    CLI_Mono -->|SSH launch| SSH
    SSH --> CLI_Client

    CLI_Client -->|polls REST| REST_Endpoints
    REST_Server -->|exposes| REST_Endpoints
    REST_Endpoints -->|queues tasks| DictDB
    REST_Endpoints -->|queues tasks| SQLite3
    REST_Endpoints -->|queues tasks| MySQL
    REST_Endpoints -->|factory| DBFactory
    REST_Endpoints -->|reads| Parameter_Files

    CLI_Client -->|runs| PolySelect
    CLI_Client -->|runs| Sieve
    CLI_Client -->|runs| Filter
    CLI_Client -->|runs| LinAlg
    CLI_Client -->|runs| Sqrt
    CLI_Client -->|writes results| REST_Endpoints

    TaskLauncher -->|dispatch| PolySelect
    TaskLauncher -->|dispatch| Sieve
    TaskLauncher -->|dispatch| Filter
    TaskLauncher -->|dispatch| LinAlg
    TaskLauncher -->|dispatch| Sqrt

    PolySelect -->|uses| GF2X
    Sieve -->|uses| Utils
    Filter -->|uses| Utils
    LinAlg -->|uses| Utils
    Sqrt -->|uses| Utils

    CI -->|build/test| CMakeLists.txt
    CI -->|build/test| Makefile
    CI -->|build/test| .gitlab-ci.yml

    Docker -->|wraps| CLI_Mono
    Docker -->|wraps| CLI_Client
    Docker -->|wraps| REST_Server

    %% Click Events
    click CLI_Mono "https://github.com/esurharun/cado-nfs/blob/main/cado-nfs.py"
    click CLI_Client "https://github.com/esurharun/cado-nfs/blob/main/cado-nfs-client.py"
    click REST_Server "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/api_server.py"
    click TopLevel "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/toplevel.py"
    click CadoTask "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/cadotask.py"
    click CadoCommand "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/cadocommand.py"
    click CadoParams "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/cadoparams.py"
    click CadoPrograms "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/cadoprograms.py"
    click DictDB "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/database/dictdb.py"
    click SQLite3 "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/database/sqlite3.py"
    click MySQL "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/database/mysql.py"
    click DBFactory "https://github.com/esurharun/cado-nfs/blob/main/scripts/cadofactor/database/factory.py"
    click Parameter_Files "https://github.com/esurharun/cado-nfs/tree/main/parameters/"
    click PolySelect "https://github.com/esurharun/cado-nfs/tree/main/polyselect/"
    click Sieve "https://github.com/esurharun/cado-nfs/tree/main/sieve/"
    click Filter "https://github.com/esurharun/cado-nfs/tree/main/filter/"
    click LinAlg "https://github.com/esurharun/cado-nfs/tree/main/linalg/"
    click Sqrt "https://github.com/esurharun/cado-nfs/tree/main/sqrt/"
    click GF2X "https://github.com/esurharun/cado-nfs/tree/main/gf2x/"
    click Utils "https://github.com/esurharun/cado-nfs/tree/main/utils/"

    %% Styles
    classDef blue fill:#D0E6FF,stroke:#0366D6,color:#0366D6;
    classDef green fill:#DFF5D8,stroke:#28A745,color:#1E4620;
    classDef orange fill:#FFE8CC,stroke:#D46A00,color:#8A3B00;
    classDef gray fill:#F0F0F0,stroke:#999999,color:#333333;
```
