How fra150/studsar works
Project type and purpose
• StudSar is a Python-based AI semantic memory library (V3) that “remembers, links and reasons” inside a single neural network. It provides text segmentation, embedding (via Sentence Transformers), a unified in-memory store for both internal and external (RAG) segments, search (cosine similarity), and persistence in a single .pth file. It is consumed as a library (via StudSarManager and RAGConnector) and demonstrated in example scripts.
File structure
• Top-level:
• README.md, requirements.txt, studsar_neural_demo.pth
• examples/ (basic_example.py, rag_example.py) – client scripts
• src/ – core modules
• tests/ – integration tests
• src/managers/manager.py: orchestrates build, search, save/load.
• src/models/neural.py: defines the unified neural network data structures and operations.
• src/rag/rag_connector.py: loads external sources (PDF/TXT/CSV/Web/DB), segments, embeds, tags, and injects them into the same memory.
• src/utils/text.py: text segmentation utilities (spaCy fallback).
• Temporary files: tmp_ai_research.txt, demo state file.
README for architectural insights
• Describes a 7-step pipeline: Segmentation → Marker generation → Storage → Emotion/Reputation/Usage tagging → RAG integration → Search → Persistence.
• Highlights use of Sentence-Transformers (all-MiniLM-L6-v2), spaCy segmentation, PyTorch .pth persistence, and tag-based filtering.
a. Components
1. Clients (Examples / API Users)
2. StudSarManager (controller)
3. Text Segmentation (spaCy / fallback)
4. Embedding Model (Sentence-Transformers)
5. Memory Store (unified tensor + metadata: emotion, reputation, usage, RAG tags)
6. RAGConnector (external source loader & tagger)
7. Search Engine (cosine similarity, tag filters)
8. Persistence Layer (save/load .pth)
9. External Sources (PDF, TXT, CSV, Web Pages, DB Rows)
b. Data flows and relationships
• Modules under src/utils and src/models are invoked by manager; tests validate RAG integration.
c. Architectural patterns and principles
• Single‐entry orchestrator (StudSarManager) following a pipeline pattern.
• Unified memory buffer pattern: both internal and external segments share one tensor store with metadata tags.
• Modular design: segmentation, embedding, storage, retrieval, persistence separated into distinct modules.
• Tag‐based filtering to support RAG and internal memory retrieval.
d. Key technologies and frameworks
• Python 3.x, PyTorch (.pth), Sentence-Transformers (all‐MiniLM‐L6‐v2), spaCy segmentation fallback, LangChain loaders (optional for advanced RAG), pytest.