How humancentricart/memo works
This repository is a research codebase for MeMo, a language-modeling approach centered on associative memory and a memorization-first workflow. It is not organized like a production app; the architecture is dominated by three alternative implementations of the same core idea, plus notebooks for experiments and demonstrations.
1. Top-level architecture
- The repo has one standalone reference implementation, two package-based implementations, and a set of experiment notebooks/results.
- The three implementations are:
- MeMoCMM.py — a compact PyTorch-only version using direct vector/matrix operations.
- MeMoPyTorch/ — a modular PyTorch version built from neural-network components.
- MeMoHF/ — a HuggingFace-compatible version shaped like a transformer model.
2. Core conceptual pipeline
- The central flow is: text input → tokenization → embedding/representation → memorization in associative memory → retrieval / forgetting / evaluation.
- The repository is explicitly about separating a memorization phase from a later learning phase, so “memory” is the key architectural boundary.
3. Reference implementation: MeMoCMM.py
- This is the simplest implementation and likely serves as the baseline for understanding the algorithm.
- It appears to express MeMo with low-level tensor math rather than a layered model abstraction.
- For diagramming purposes, treat this as the “minimal core” version of the algorithm.
4. Modular PyTorch package: MeMoPyTorch/
- This package is the cleanest place to look for a componentized architecture.
- It contains separate modules for:
- modelling_memo.py — the model assembly/orchestration layer.
- modelling_memo_layer.py — the core memory/model layer.
- modelling_memo_embedding.py — embedding and representation logic.
- modelling_memo_tokenizer.py — text-to-token conversion.
- evaluating_memo.py — evaluation helpers.
- modelling_memo_exception.py — domain-specific error handling.
- This version is meant to be “ready for learning after memorization,” so it likely exposes a clearer interface for downstream training loops.
5. HuggingFace-compatible package: MeMoHF/
- This package mirrors the PyTorch structure but adapts MeMo to HuggingFace-style model conventions.
- It includes:
- modelling_memo.py
- modelling_memo_configuration.py
- modelling_memo_embedding.py
- modelling_memo_layer.py
- modelling_memo_tokenizer.py
- evaluating_memo.py
- modelling_memo_exception.py
- The presence of a configuration module suggests this version plugs into model/config ecosystems used by transformer tooling.
6. Shared subsystems across the packages
- The repeated file names across MeMoPyTorch and MeMoHF indicate a shared conceptual decomposition:
- tokenizer boundary
- embedding boundary
- memory/layer boundary
- model wrapper
- evaluation boundary
- These are the main architectural boxes to show in a diagram, because they define the flow of data and responsibilities.
7. Memory mechanism as the central domain boundary
- The repository’s differentiator is the associative memory mechanism, which appears to be the heart of both memorization and retrieval.
- “Forgetting” is also treated as a first-class operation, which means the model is not just a static encoder but a mutable memory system.
- In a diagram, this memory component should sit between representation generation and output/evaluation.
8. Evaluation subsystem
- Both package versions include evaluating_memo.py, implying that evaluation is implemented alongside the model rather than externally.
- This suggests the repo uses evaluation to measure memorization/retrieval behavior and probably to compare against experimental settings from the paper.
9. Notebook-based experimentation layer
- The Experiments/ directory contains notebooks and result spreadsheets, showing that much of the repo is exploratory and empirical.
- The PlayingWithMemo*.ipynb notebooks serve as demonstrations of memorizing, retrieving, and forgetting across the different implementations.
- Playground.py appears to be a script entry point for interactive experimentation outside notebooks.
10. Research workflow and outputs
- The .xlsx files in Experiments/ are output artifacts, likely recording metrics or tables from the paper.
- The repository therefore has a clear separation between:
- library code implementing MeMo
- notebook/script experimentation
- saved experimental results
11. Boundary between implementation styles
- MeMoCMM.py, MeMoPyTorch, and MeMoHF are parallel implementations rather than layers of one stack.
- They represent different integration targets:
- pure tensor prototype
- PyTorch module-based model
- HuggingFace ecosystem compatibility
- An architecture diagram should show them as sibling variants, not as dependencies stacked on one another.
12. External ecosystem relevance
- The only materially important external ecosystems are PyTorch and HuggingFace, because they shape the package structure and intended reuse.
- There are no signs of a backend service, database, or API infrastructure; this is primarily a model-research repository.
13. What to emphasize in a diagram
- Put tokenizer → embedding → memory/layer → model wrapper → evaluation as the core flow.
- Surround that with three implementation variants: CMM, PyTorch, HuggingFace.
- Show notebooks/experiments as consumers of the library rather than as part of the runtime model path.
14. Overall characterization
- This repo is best understood as a research lab for associative-memory language models, with multiple implementation tiers and a strong emphasis on experimentation.
- The most important architectural idea is the separation of memorization mechanics from model integration, which is reflected consistently across the code organization.
Main entry points: MeMoPyTorch/modelling_memo.py Model wrapper · MeMoHF/modelling_memo.py HF model · modelling_memo_tokenizer.py Tokenizer · MemorizationExperiments.ipynb Notebooks
Components 18
Implementations
Parallel MeMo implementations for different integration targets.
- MeMo CMM · Compact PyTorch-only baseline using direct tensor operations.
MeMoCMM.py - Model wrapper · Assembles the modular PyTorch MeMo pipeline.
MeMoPyTorch/modelling_memo.py - Memory layer · Core memorization, retrieval, and forgetting logic for the PyTorch package.
MeMoPyTorch/modelling_memo_layer.py - Embedding · Converts tokens into internal representations for memorization.
MeMoPyTorch/modelling_memo_embedding.py - Tokenizer · Text-to-token boundary for the modular PyTorch implementation.
MeMoPyTorch/modelling_memo_tokenizer.py - Evaluation · Evaluation helpers for memorization and retrieval behavior.
MeMoPyTorch/evaluating_memo.py - Exceptions · Domain-specific errors for the PyTorch implementation.
MeMoPyTorch/modelling_memo_exception.py - HF model · HuggingFace-style MeMo wrapper aligned with model conventions.
MeMoHF/modelling_memo.py - Config · Model configuration for HuggingFace integration.
MeMoHF/modelling_memo_configuration.py - Memory layer · Core MeMo memory layer adapted to HuggingFace packaging.
MeMoHF/modelling_memo_layer.py - Embedding · Representation logic for the HuggingFace-compatible variant.
MeMoHF/modelling_memo_embedding.py - Tokenizer · Text-to-token boundary for the HuggingFace-compatible variant.
MeMoHF/modelling_memo_tokenizer.py - Evaluation · Evaluation helpers for the HuggingFace-compatible variant.
MeMoHF/evaluating_memo.py - Exceptions · Domain-specific errors for the HuggingFace-compatible variant.
MeMoHF/modelling_memo_exception.py
Research
Notebooks, scripts, and saved experiment outputs.
- Notebooks · Interactive memorization and retrieval demonstrations.
Experiments/MemorizationExperiments.ipynb - Results · Saved experiment tables and metrics.
Experiments/MemorizationResults.xlsx - Playground · Interactive script for trying MeMo workflows outside notebooks.
Playground.py - Demo notebooks · Notebook demonstrations across the PyTorch and HF variants.
PlayingWithMemo.ipynb
Connections 20
- MeMo CMM → Tokenizer: conceptual flow · The baseline expresses the same text-to-memory pipeline in minimal tensor form.
- Tokenizer → Embedding: tokens · Tokenized text is converted into internal representations.
- Embedding → Memory layer: representations · Representations are stored and manipulated by associative memory.
- Memory layer → Model wrapper: state · The model wrapper coordinates memory behavior and learning workflow.
- Model wrapper → Evaluation: evaluate · Model outputs and memory behavior are measured through evaluation helpers.
- Tokenizer → Model wrapper: input · Tokenizer feeds the assembled PyTorch model.
- Exceptions → Model wrapper: errors · Domain exceptions support the PyTorch model boundary.
- Config → HF model: configure · Configuration drives HuggingFace model construction.
- Tokenizer → Embedding: tokens · Tokenized inputs become representations.
- Embedding → Memory layer: representations · Representations are routed into associative memory.
- Memory layer → HF model: state · The HF model wrapper coordinates memory and transformer-style integration.
- HF model → Evaluation: evaluate · Evaluation measures memorization, retrieval, and forgetting behavior.
- Exceptions → HF model: errors · Domain exceptions support the HF model boundary.
- HF model → Model wrapper: parallel variant · The HF package mirrors the modular PyTorch architecture with ecosystem-specific packaging.
- Notebooks → Model wrapper: uses · Notebooks consume the modular implementation for experiments.
- Notebooks → HF model: uses · Notebooks also compare the HuggingFace-compatible variant.
- Demo notebooks → Model wrapper: demonstrates · Demo notebooks exercise memorization and forgetting workflows.
- Demo notebooks → MeMo CMM: demonstrates · Some notebooks explore the reference implementation.
- Playground → Model wrapper: drives · The playground script provides an interactive entry point.
- Results → Notebooks: outputs · Spreadsheets record the outcomes of notebook-driven experiments.