How esurharun/cado-nfs works
Project type and purpose
• Cado-NFS is an open-source, command-line driven toolset implementing the Number Field Sieve algorithm to factor large integers.
• It comprises two major “flavors” of interface:
a. A monolithic CLI (cado-nfs.py) that orchestrates all steps locally or via SSH.
b. A client/server mode (under scripts/cadofactor/) where a Python-based server dispatches work to independent clients.
• Under the hood, a collection of C/C++ binaries cover each stage: polynomial selection, sieving, filtering, linear algebra, square root, plus support libraries (GF2X, linear algebra, utilities).
Key directories and their roles
• / (root) – orchestration scripts, top-level Makefile/CMake, entry-point cado-nfs.py, cado-nfs-client.py.
• config/, .gitlab-ci.yml, ci/ – build-system checks, platform-detection tests, CI pipelines.
• gf2x/ – stand-alone library for GF(2^n) arithmetic.
• linalg/ – linear algebra routines (including parallel MPI/OpenMP implementations).
• polyselect/ – polynomial selection code and Sage bindings.
• sieve/ – lattice sieving code, ECM & bucket sieving.
• filter/ – relation filtering and logging stages.
• sqrt/ – square-root CRT and specialized routines.
• utils/ + misc/ – shared utilities (memory managers, multi-precision support, thread pools, JSON, etc.).
• scripts/ – Python scripts:
• cadofactor/ – server, client, database backends (dictdb, MySQL), REST API (api_server.py).
• descent/ – algebraic descent helpers.
• build, lint, packaging tools, Docker wrappers.
• parameters/ – default parameter sets for various factorization scenarios.
• tests/ – unit and integration tests for each component.
System components
A. User Interface
• cado-nfs.py CLI
• scripts/cadofactor/api_server.py (Flask-based REST server)
• cado-nfs-client.py CLI for client mode
B. Orchestration Layer (Python)
• Parameter parser → decides on local vs. distributed execution
• Task launcher → spawns processes or SSH commands
• In server mode, REST endpoints manage workunit lifecycle
C. Data Storage
• In-memory/dictdb (default) or persistent storage via SQLite/MySQL
• Parameter files and snapshots under parameters/
D. Compute Layer (binaries)
1. polyselect (C/C++ + Sage glue)
2. sieve (C++ sieving engines + ECM)
3. filter (C code for relation filtering)
4. linalg (C++ MPI/OpenMP linear algebra modules)
5. sqrt (C code for square root step)
E. Utility Libraries
• GF2X library
• utils (multi-precision, threading, memory, JSON, etc.)
F. External Services & Tools
• SSH for remote client launch
• Docker (optional containers)
• CI pipelines (.gitlab-ci.yml)
• Third-party libs: GMP, MPIR, hwloc, MPI, OpenMP
Interactions and data flow
• User runs cado-nfs.py → orchestration layer reads parameter files →
• Local mode: directly invokes compute binaries in sequence or in parallel.
• SSH mode: uses SSH to launch cado-nfs-client.py on remote hosts.
• Server mode: Python REST server enqueues tasks in DB → clients poll REST API → fetch workunits → run binaries → post results back.
• Each compute stage reads/writes intermediate files under the work directory.
• Filtering and linear algebra may exchange data via files or MPI.
Architectural patterns & principles
• Modular pipeline: each algorithmic step is encapsulated in its own binary.
• Layered orchestration: Python script sits atop the C/C++ compute layer.
• Adapter pattern: multiple DB backends, multiple scheduling modes (local, SSH, REST).
• Plugin/extensibility points: parameter files, strategy scripts under sieve/strategies.
• Build-time configurability via CMake and Make abstractions.
Essential details to capture
• Separation of concerns: orchestration vs. compute
• Multiple execution modes (local, SSH, REST)
• Data persistence (parameter files, database)
• Use of parallelism (threads, MPI)
• Extensibility (strategies, parameter sets)