# beyonddunbar/fasttext architecture

> Architecture diagram of the GitHub repository beyonddunbar/fasttext, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/beyonddunbar/fasttext
- Repository: https://github.com/beyonddunbar/fasttext

## Overview

##### Project type and purpose
- fastText is a command-line and library toolkit for learning word embeddings and text classifiers.
- It is implemented in C++, with a CLI front-end (`fasttext` binary), Python bindings, a WebAssembly build, example scripts, and documentation.

##### File structure
- Top-level components:
- **src/**: Core C++ implementation of models (dictionary, matrix, loss, quantization, etc.)
- **Makefile, CMakeLists.txt, python/ subdir**: Build orchestration for C++ binary and Python module
- **alignment/**, **crawl/**, **scripts/**: Example workflows and auxiliary tools (data download, preprocessing, evaluation)
- **python/fasttext_module/**: Pybind11-based Python binding to the C++ core
- **webassembly/**: Emscripten-compiled C++ to WASM & JS glue for in-browser inference
- **website/**: Static site sources and generated docs (React, Docusaurus, static HTML)
- **docs/**: Markdown documentation for features, options, tutorials
- **.circleci/**: CI pipelines for testing builds across C++, Python, and environments

##### Extract additional insights from the README
- Core features: skipgram/cbow word-vector training, supervised text classification, quantization
- Multiple build targets: CLI binary, shared/static libraries, Python wheel, WebAssembly module
- External dependencies: C++11 compiler, make/CMake, Python 2.7/3.x + NumPy/SciPy, pybind11, Emscripten (for WASM)

##### Main components
a. **Core C++ Engine**
- Modules: Dictionary, Matrix, Loss, Model, ProductQuantizer, Utils
- Build outputs: `fasttext` CLI binary, static/shared library, WASM module
b. **CLI Layer**
- `fasttext` executable (built from `src/main.cc`)
- Parses user arguments, invokes engine APIs
c. **Python Binding**
- pybind11 wrapper (`fasttext_pybind.cc`) → Python package `fasttext`
- Exposes core APIs to Python scripts and notebooks
d. **WebAssembly Module**
- Emscripten build of core engine (`fasttext_wasm.cc`) → `fasttext.wasm` + `fasttext.js`
- Used by browser examples (`webassembly/doc/examples/*.html`)
e. **Example Workflows & Scripts**
- alignment/ unsupervised/supervised scripts, crawl tools, classification examples
- Shell and C++ helpers to download data, process, train, eval
f. **CI/CD Pipeline**
- CircleCI config → runs CMake, Make, Python tests, pip tests, linting
g. **Documentation & Website**
- docs/ (Markdown) → docs site via Docusaurus (`website/`)
- Static HTML API docs (Doxygen output in `website/static/docs/en/html`)

##### Relationships and interactions
- Build system (Makefile/CMake) produces:
- CLI binary → used by example scripts & end-users
- Python wheel → used by Python examples and third-party code
- WASM module → consumed by web pages & in-browser demos
- Example scripts call CLI or Python API to train/evaluate models
- Website serves documentation and WebAssembly demos
- CircleCI orchestrates building & testing of all targets

##### Architectural patterns and design principles
- **Modular core**: engine logic separated into self-contained C++ classes
- **Multi-target builds**: same core compiled for native, Python, WASM
- **Separation of concerns**: core engine vs. language bindings vs. examples vs. docs vs. website
- **Monorepo**: all artifacts in one repository, CI covers all

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Global Users
    U1["Shell/User"]:::external
    U2["Python/User"]:::external
    U3["Browser/User"]:::external

    %% Build System
    Make["Makefile"]:::build
    CMake["CMakeLists.txt"]:::build
    subgraph "Build System"
        Make
        CMake
    end

    %% Core C++ Engine
    subgraph "Core C++ Engine"
        CE1["Dictionary"]:::core
        CE2["Matrix"]:::core
        CE3["Loss"]:::core
        CE4["Model"]:::core
        CE5["ProductQuantizer"]:::core
        CE6["Utils"]:::core
    end

    %% CLI Layer
    subgraph "Native CLI (C++)"
        CLI["fasttext CLI"]:::core
    end

    %% Python Binding
    subgraph "Python Package" 
        PY1["pybind11 Layer"]:::bindings
        PY2["fasttext Python API"]:::bindings
    end

    %% WebAssembly Module
    subgraph "WebAssembly Module"
        WASM1["fasttext.wasm/js"]:::bindings
    end

    %% Example Workflows & Scripts
    subgraph "Example Workflows & Scripts"
        EX1["alignment/"]:::examples
        EX2["crawl/"]:::examples
        EX3["scripts/"]:::examples
    end

    %% Documentation & Website
    subgraph "Documentation"
        DOCS["Markdown Docs"]:::docs
    end
    subgraph "Website"
        WS1["Docusaurus Site"]:::docs
    end

    %% CI/CD Pipeline
    subgraph "CI/CD (CircleCI)"
        CI[".circleci/config.yml"]:::cicd
    end

    %% Build Flows
    Make -->|"builds"| CE1
    CMake -->|"builds"| CE1
    Make -->|"builds"| CLI
    CMake -->|"builds"| CLI
    Make -->|"builds"| PY1
    CMake -->|"builds"| PY1
    Make -->|"builds"| WASM1
    CMake -->|"builds"| WASM1

    %% Runtime Flows
    U1 -->|"invokes"| CLI
    U2 -->|"imports"| PY2
    PY2 -->|"calls"| PY1
    PY1 -->|"uses core"| CE1
    U3 -->|"loads"| WASM1
    WASM1 -->|"uses core"| CE1

    %% Core Module Composition
    CE1 --> CE2
    CE1 --> CE3
    CE1 --> CE4
    CE1 --> CE5
    CE1 --> CE6

    %% Examples usage
    EX1 -->|"calls CLI"| CLI
    EX1 -->|"imports PY API"| PY2
    EX2 -->|"calls CLI"| CLI
    EX3 -->|"calls CLI"| CLI

    %% Docs to Website
    DOCS -->|"generates"| WS1

    %% CI/CD connections
    CI -->|"runs build/tests"| Make
    CI -->|"runs build/tests"| CMake

    %% Click Events
    click Make "https://github.com/beyonddunbar/fasttext/tree/main/Makefile"
    click CMake "https://github.com/beyonddunbar/fasttext/blob/main/CMakeLists.txt"
    click CE1 "https://github.com/beyonddunbar/fasttext/tree/main/src/"
    click CLI "https://github.com/beyonddunbar/fasttext/blob/main/src/main.cc"
    click PY1 "https://github.com/beyonddunbar/fasttext/blob/main/python/fasttext_module/fasttext/pybind/fasttext_pybind.cc"
    click EX1 "https://github.com/beyonddunbar/fasttext/tree/main/alignment/"
    click EX2 "https://github.com/beyonddunbar/fasttext/tree/main/crawl/"
    click EX3 "https://github.com/beyonddunbar/fasttext/tree/main/scripts/"
    click CI "https://github.com/beyonddunbar/fasttext/blob/main/.circleci/config.yml"
    click DOCS "https://github.com/beyonddunbar/fasttext/tree/main/docs/"
    click WS1 "https://github.com/beyonddunbar/fasttext/tree/main/website/"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0177FF
    classDef bindings fill:#DFFFD0,stroke:#77BF00
    classDef examples fill:#FFE8C0,stroke:#FF9000
    classDef docs fill:#F2F2F2,stroke:#666
    classDef cicd fill:#E8D0FF,stroke:#7700BF
    classDef build fill:#FFF2CC,stroke:#BF9000
    classDef external stroke-dasharray: 4 2
```
