How nvidia-ai-blueprints/transaction-foundation-model works
1. Repository purpose and shape
This is a notebook-driven ML research/example repo for building a transaction foundation model. The architecture is a pipeline: raw financial transaction tables are converted into token sequences, used to pretrain a decoder-only model, then repurposed as embeddings for fraud detection and visualization.
2. Primary execution entrypoints
The main user-facing flow lives in the five notebooks at the repo root. They are intended to be run in order:
- dataset baseline
- tokenizer construction
- foundation-model training
- embedding extraction / visualization
- downstream fraud classification
The notebooks are the orchestration layer; the src/ and scripts/ code provide reusable components underneath.
3. Data ingestion and baseline branch
01_dataset_baseline.ipynb loads the TabFormer financial transaction dataset, performs temporal train/validation/test splitting, and trains an XGBoost fraud baseline on raw/tabular features. This establishes the non-foundation-model comparison branch early in the pipeline.
4. Tokenization subsystem
src/tokenizer/ is the main preprocessing subsystem. It is modular and schema-aware, with components for:
- numerical fields
- categorical hashing
- fixed-vocabulary mappings
- time-delta encoding
- pipeline composition for financial records
The key architectural point is that heterogeneous transaction columns are transformed into a domain-specific token vocabulary rather than generic text tokens.
5. GPU-accelerated preprocessing boundary
The README makes it clear that tokenization is designed to run with NVIDIA RAPIDS / cuDF / CuPy. That means the preprocessing boundary is not just “Python data prep”; it is a GPU-oriented transformation stage that turns tabular rows into sequence model inputs at scale.
6. Sequence modeling data prep
src/clm_data.py sits between tokenized records and model training. Its role is to package the token sequences for causal language modeling so the decoder can learn next-token prediction over transaction histories rather than supervised fraud labels.
7. Foundation model training subsystem
scripts/train_decoder_model.py plus configs/pretrain_financial_decoder.yaml define the training path. The model is a decoder-only transformer trained with NVIDIA NeMo AutoModel using HuggingFace-compatible architecture settings. The README specifies the example uses a small Llama-style decoder, trained for causal LM on GPU.
8. Checkpoint and artifact boundary
The reusable pretrained checkpoint lives in models/decoder-foundation-model/ and is tracked with Git LFS. This is a distinct artifact from the notebook demo training output (models/decoder-demo/), which is not the checkpoint used by later notebooks. That separation matters for the architecture: one path demonstrates training, another path consumes the shipped model artifact.
9. Inference and embedding extraction
src/decoder_inference.py loads the pretrained decoder and runs GPU inference. The downstream representation is produced by last-token pooling, yielding fixed-size embeddings (the README calls out 512-dimensional vectors). This is the bridge from sequence modeling to classical ML workflows.
10. Downstream analytics branch
04_inference_embedding_extraction.ipynb visualizes embeddings with UMAP, while 05_xgboost_fraud_detection.ipynb evaluates fraud detection using:
- raw features
- foundation-model embeddings
- combined features
This shows the intended reuse pattern: the foundation model is not the final predictor; it is a representation learner feeding classical ML.
11. Runtime and infrastructure context
The repo is designed to run inside the NVIDIA NeMo Framework container with CUDA-enabled GPUs. The notebooks expect the containerized environment and rely on inline-installed notebook dependencies. This is an offline research workflow, not a deployed service.
12. What the architecture diagram should emphasize
The clearest diagram would show two converging paths:
- tabular transactions → tokenizer → decoder pretraining → checkpoint → embeddings
- raw tabular features → XGBoost baseline / downstream classifier
The shared boundary is the dataset; the key reusable artifact is the pretrained decoder checkpoint; the main value-add is that learned embeddings improve fraud detection over hand-crafted features.
Main entry points: financial_tokenizer.py Financial tokenizer · train_decoder_model.py Train script · 02_seq_preproc_tokenization.ipynb Tokenization · 04_inference_embedding_extraction.ipynb Extract embeddings
Components 19
Notebook flow
User-facing orchestration for dataset prep, training, inference, and evaluation.
- Dataset baseline · Loads transactions, performs temporal splits, and trains a raw-feature fraud baseline.
01_dataset_baseline.ipynb - Tokenization · Drives schema-aware transaction tokenization and sequence preparation.
02_seq_preproc_tokenization.ipynb - Train decoder · Launches foundation-model pretraining for the decoder-only transformer.
03_foundation_model_training.ipynb - Extract embeddings · Runs pretrained-model inference to produce embeddings and visualize them.
04_inference_embedding_extraction.ipynb - Fraud eval · Compares raw features, embeddings, and combined features for fraud detection.
05_xgboost_fraud_detection.ipynb
Tokenization
GPU-oriented preprocessing that converts transaction tables into domain tokens.
- Token pipeline · Composes the column-wise transaction tokenization stages.
src/tokenizer/pipeline.py - Financial tokenizer · Schema-aware tokenizer for heterogeneous financial records.
src/tokenizer/financial_tokenizer.py - Financial pipeline · Higher-level preprocessing flow for financial transaction data.
src/tokenizer/financial_pipeline.py - Numeric tokens · Encodes numerical transaction fields into token form.
src/tokenizer/numerical.py - Category hashing · Hashes categorical fields into a compact token space.
src/tokenizer/categorical_hash.py - Fixed vocab · Maps selected fields to fixed vocabulary tokens.
src/tokenizer/fixed_vocab.py - Time deltas · Encodes temporal gaps between transactions.
src/tokenizer/timedelta.py
Modeling
Causal language modeling training and inference for the decoder-only foundation model.
- CLM data · Packages tokenized transaction histories for causal language modeling.
src/clm_data.py - Train script · Python entrypoint that launches decoder pretraining.
scripts/train_decoder_model.py - Pretrain config · Model and optimization settings for decoder pretraining.
configs/pretrain_financial_decoder.yaml - Decoder inference · Loads the pretrained decoder and produces sequence representations.
src/decoder_inference.py
Artifacts
Persisted pretrained checkpoint consumed by later notebooks.
- Decoder checkpoint · Persisted pretrained decoder model reused by inference and downstream analysis.
Downstream
Embedding visualization and fraud classification on top of learned representations.
- Embeddings · Fixed-size last-token pooled vectors used for visualization and classifiers.
- Fraud model · XGBoost fraud detection on raw, learned, and combined features.
Connections 21
- Dataset baseline → Tokenization: dataset flow · Temporal split and baseline data preparation feed the tokenization stage.
- Tokenization → Financial pipeline: uses · Notebook orchestrates the financial tokenization pipeline.
- Financial pipeline → Token pipeline: composes · Higher-level financial preprocessing is built from the generic pipeline.
- Token pipeline → Financial tokenizer: drives · Pipeline applies the schema-aware financial tokenizer.
- Financial tokenizer → Numeric tokens: encodes · Numeric transaction columns are converted to tokens.
- Financial tokenizer → Category hashing: encodes · Categorical columns are hashed into token ids.
- Financial tokenizer → Fixed vocab: encodes · Selected fields are mapped through a fixed vocabulary.
- Financial tokenizer → Time deltas: encodes · Relative time gaps become temporal tokens.
- Tokenization → CLM data: prepares · Tokenized records are packed for causal language modeling.
- Train decoder → Train script: launches · Notebook delegates training to the script entrypoint.
- Train decoder → Pretrain config: configures · Training notebook uses the decoder pretraining configuration.
- Train script → CLM data: consumes · Training reads sequence-packed CLM data.
- Train script → Decoder checkpoint: produces · Pretraining writes the reusable decoder checkpoint artifact.
- Pretrain config → Train script: sets · Config specifies architecture and optimization settings.
- Decoder checkpoint → Extract embeddings: loads · Embedding extraction notebook consumes the pretrained checkpoint.
- Extract embeddings → Decoder inference: uses · Notebook runs the inference runtime for pooled representations.
- Decoder inference → Embeddings: outputs · Inference produces fixed-size transaction embeddings.
- Embeddings → Fraud eval: feeds · Learned representations are evaluated in downstream fraud analysis.
- Dataset baseline → Fraud model: baseline · Raw transaction features support the initial fraud baseline.
- Fraud eval → Fraud model: evaluates · Classifier compares raw, embedding, and combined feature sets.
- Extract embeddings → Fraud eval: hand off · Embedding extraction results are reused by fraud evaluation.