How beyonddunbar/fasttext works
Project type and purpose
• fastText is a command-line and library toolkit for learning word embeddings and text classifiers.
• It is implemented in C++, with a CLI front-end (fasttext binary), Python bindings, a WebAssembly build, example scripts, and documentation.
File structure
• Top-level components:
• src/: Core C++ implementation of models (dictionary, matrix, loss, quantization, etc.)
• Makefile, CMakeLists.txt, python/ subdir: Build orchestration for C++ binary and Python module
• alignment/, crawl/, scripts/: Example workflows and auxiliary tools (data download, preprocessing, evaluation)
• python/fasttext_module/: Pybind11-based Python binding to the C++ core
• webassembly/: Emscripten-compiled C++ to WASM & JS glue for in-browser inference
• website/: Static site sources and generated docs (React, Docusaurus, static HTML)
• docs/: Markdown documentation for features, options, tutorials
• .circleci/: CI pipelines for testing builds across C++, Python, and environments
Extract additional insights from the README
• Core features: skipgram/cbow word-vector training, supervised text classification, quantization
• Multiple build targets: CLI binary, shared/static libraries, Python wheel, WebAssembly module
• External dependencies: C++11 compiler, make/CMake, Python 2.7/3.x + NumPy/SciPy, pybind11, Emscripten (for WASM)
Main components
a. Core C++ Engine
• Modules: Dictionary, Matrix, Loss, Model, ProductQuantizer, Utils
• Build outputs: fasttext CLI binary, static/shared library, WASM module
b. CLI Layer
• fasttext executable (built from src/main.cc)
• Parses user arguments, invokes engine APIs
c. Python Binding
• pybind11 wrapper (fasttext_pybind.cc) → Python package fasttext
• Exposes core APIs to Python scripts and notebooks
d. WebAssembly Module
• Emscripten build of core engine (fasttext_wasm.cc) → fasttext.wasm + fasttext.js
• Used by browser examples (webassembly/doc/examples/*.html)
e. Example Workflows & Scripts
• alignment/ unsupervised/supervised scripts, crawl tools, classification examples
• Shell and C++ helpers to download data, process, train, eval
f. CI/CD Pipeline
• CircleCI config → runs CMake, Make, Python tests, pip tests, linting
g. Documentation & Website
• docs/ (Markdown) → docs site via Docusaurus (website/)
• Static HTML API docs (Doxygen output in website/static/docs/en/html)
Relationships and interactions
• Build system (Makefile/CMake) produces:
• CLI binary → used by example scripts & end-users
• Python wheel → used by Python examples and third-party code
• WASM module → consumed by web pages & in-browser demos
• Example scripts call CLI or Python API to train/evaluate models
• Website serves documentation and WebAssembly demos
• CircleCI orchestrates building & testing of all targets
Architectural patterns and design principles
• Modular core: engine logic separated into self-contained C++ classes
• Multi-target builds: same core compiled for native, Python, WASM
• Separation of concerns: core engine vs. language bindings vs. examples vs. docs vs. website
• Monorepo: all artifacts in one repository, CI covers all