# inseefrlab/torch-fasttext architecture

> Architecture diagram of the GitHub repository inseefrlab/torch-fasttext, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/inseefrlab/torch-fasttext
- Repository: https://github.com/inseefrlab/torch-fasttext

## Overview

##### Project Type and Purpose
- The project is a specialized machine learning library for text classification that leverages PyTorch and PyTorch Lightning.
- It is designed to build, train, predict, and explain FastText-based models. Support for extra features like n-gram tokenization and flexible handling of categorical features (with various merging strategies) is a crucial aspect.
- The README shows sample usage, installation, and references to various FastText architectures.

##### Analyzing the File Structure
- Top-level directories include configuration (e.g., .github for CI/CD workflows), tests, notebooks (examples and experiments), and the main package directory (torchFastText).
- Within the torchFastText package, subdirectories are clearly aligned with major functionalities:
- datasets: Contains code for managing datasets and tokenization.
- model: Contains model architecture definitions (the PyTorch Lightning module, loss functions and the underlying PyTorch implementations).
- preprocess: Contains text preprocessing logic.
- utilities: Contains helper functions, checkers, and general utilities.
- Presence of build configurations (pyproject.toml, requirements.txt) along with tests indicates a well-packaged library.

##### Core Components & Relationships
a. Main Components to Represent:
- TorchFastText Core Library, which serves as the primary interface (note the torchFastText.py file).
- Model Component: Includes the PyTorch model, Lightning module, and losses. This is central for training, prediction, and model evaluations.
- Dataset Component: Responsible for loading and preparing datasets, and includes tokenization (n-gram tokenization) logic.
- Preprocessing: Handles the text and categorical data preprocessing, ensuring inputs are in a proper format.
- Explainability Component: Provides model explanation capabilities and visualization (leverages libraries like Captum).
- Utilities: Contains additional internal checks and helper routines.
- Testing: The tests directory represents a separate validation layer ensuring correctness.
- CI/CD: GitHub workflows for continuous integration and publishing add a peripheral component for deployment and maintenance.
- Notebooks: Serve as demonstration and documentation for users, linking usage examples with the library.

b. Relationships and Interactions:
- The primary user interface (torchFastText) depends on preprocessing, datasets, and model creation.
- After preprocessing, the prepared data feeds into the model for training via the Lightning module.
- Post-training, predictions can be generated and optionally explained using the explanation submodule.
- Utility functions are used across modules for error checking, data handling, etc.
- External dependencies like PyTorch Lightning and NumPy should be noted, as they underpin the training loop and tensor operations.

c. Important Architectural Patterns/Design Principles:
- Modularity: Separate subpackages (model, datasets, preprocess, explainability) allow for extension and easy testing.
- Dependency Injection: Components (e.g., optimizer and scheduler configurations) are meant to be configurable.
- Clear separation of concerns, with distinct layers ensuring data is preprocessed, fed into a model, and then evaluated or explained.
- Pipeline for categorical features: Different embedding combinations (summing vs. averaging vs. concatenation) are handled as part of the model architecture.

d. Relevant Technologies/Frameworks:
- PyTorch and PyTorch Lightning: For model creation and training logic.
- NumPy: Likely used for data manipulation.
- Optional libraries: nltk, unidecode for preprocessing, Captum for explainability.
- CI/CD Tools: GitHub Actions for testing and publishing.

##### Specific Elements
- Clear Labels: Each major component (Preprocessing, Datasets, Model, Explainability, Utilities) should be clearly labeled.
- Optional Dependencies: Mark components (like explainability and preprocessing) which can be optionally installed.
- CI/CD and Testing: Represent these as separate pipelines ensuring code quality and automated deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Pipeline
    subgraph "Core Pipeline"
        TF["TorchFastText Core Library"]:::core
        PRE["Preprocessing Module"]:::module
        DS["Dataset Component"]:::module
        MD["Model Component (sum/average/concatenate)"]:::model
        PRD["Prediction Output"]:::module
        EXPL["Explainability Module"]:::expl

        TF -->|"calls"| PRE
        PRE -->|"prepares"| DS
        DS -->|"feeds"| MD
        MD -->|"generates"| PRD
        MD -->|"explains"| EXPL
    end

    %% External Dependencies
    subgraph "External Dependencies"
        PTL["PyTorch Lightning"]:::external
        PT["PyTorch"]:::external
        NP["NumPy"]:::external
    end

    MD -->|"utilizes"| PTL
    MD -->|"utilizes"| PT
    MD -->|"utilizes"| NP

    %% Support Components
    subgraph "Support Components"
        UT["Utilities Module"]:::support
        TST["Testing Framework"]:::support
        CICD["CI/CD Pipeline"]:::support
        NB["Usage Examples & Notebooks"]:::support
    end

    TF -->|"uses"| UT
    PRE -->|"gets support from"| UT
    DS -->|"gets support from"| UT
    MD -->|"gets support from"| UT
    EXPL -->|"gets support from"| UT

    %% Linking support components amongst themselves
    NB --> TST
    TST --> CICD

    %% Click Events
    click TF "https://github.com/inseefrlab/torch-fasttext/blob/main/torchFastText/torchFastText.py"
    click PRE "https://github.com/inseefrlab/torch-fasttext/blob/main/torchFastText/preprocess/preprocess.py"
    click DS "https://github.com/inseefrlab/torch-fasttext/tree/main/torchFastText/datasets"
    click MD "https://github.com/inseefrlab/torch-fasttext/tree/main/torchFastText/model"
    click EXPL "https://github.com/inseefrlab/torch-fasttext/blob/main/torchFastText/explainability/visualisation.py"
    click UT "https://github.com/inseefrlab/torch-fasttext/tree/main/torchFastText/utilities"
    click TST "https://github.com/inseefrlab/torch-fasttext/tree/main/tests"
    click CICD "https://github.com/inseefrlab/torch-fasttext/tree/main/.github/workflows"
    click NB "https://github.com/inseefrlab/torch-fasttext/tree/main/notebooks"

    %% Styles
    classDef core fill:#6ba7ff,stroke:#333,stroke-width:2px;
    classDef module fill:#a3c4f3,stroke:#333,stroke-width:2px;
    classDef model fill:#81c784,stroke:#333,stroke-width:2px;
    classDef expl fill:#fdd835,stroke:#333,stroke-width:2px;
    classDef external fill:#ffcc80,stroke:#333,stroke-width:2px;
    classDef support fill:#ce93d8,stroke:#333,stroke-width:2px;
```
