How nath1295/llmflex works
Project type and purpose
• LLMFlex is an open-source Python library (CLI tool and Streamlit frontend) for building AI applications on local and remote large language models (LLMs).
• It provides modular abstractions for LLM loading/generation, embeddings, vector databases, tools, agents, memory, prompt templates, and a web UI.
File structure
• src/llmflex – the core library, organized into subpackages:
• Agents – orchestrators combining LLMs and tools
• Embeddings – embedding model toolkits
• Frontend – Streamlit interface and app resources
• KnowledgeBase – document loaders and retrieval wrappers
• Memory – chat memory classes (short-term, long-term via FAISS)
• Models – LLM “cores” (OpenAI, HuggingFace, llama-cpp, ExLlamaV2) + LlmFactory
• Prompts – PromptTemplate formatters
• Rankers – rerankers for search results
• Schemas – Pydantic document/token schemas
• TextSplitters – token-based text splitting
• Tools – BaseTool subclasses (browser tool, SD tool, web-search utils)
• VectorDBs – BaseVectordb and FaissVectorDatabase
• cli.py – entry point for CLI commands (interface, create-app-config)
• pyproject.toml – build/config metadata
• docs/ & docs/markdown – HTML/Markdown API documentation
• imgs – UI screenshots
README for additional insights
• Key dependencies: PyTorch, llama-cpp-python, optional ExLlamaV2, AutoGPTQ/AWQ, FAISS
• Two user front ends:
• CLI commands (e.g. llmflex interface)
• Streamlit webapp for chatting with memory, RAG, function calling
• Core features: LlmFactory, LangChain compatibility, embedding toolkits, RAG via Faiss, chat memories, agents, tools, prompt templates.
a. Main components
• User Interfaces: CLI layer, Streamlit Frontend
• Control/Orchestration: Agents, CLI dispatcher
• Model Layer: LlmFactory, Model Cores (OpenAICore, HuggingFaceCore, LlamaCppCore, ExllamaCore)
• Embedding Layer: EmbeddingsToolkit (HuggingFace/API), TextSplitters
• Data Layer: VectorDB (FaissVectorDatabase), Memory (BaseMemory, LongShortTermMemory), KnowledgeBase
• Tools Layer: BrowserTool, SDTool, ToolUtils, WebSearchUtils
• Prompt Layer: PromptTemplate
• External Services: OpenAI API servers, HuggingFace Hub, DuckDuckGo, local GPU drivers, FAISS native library
b. Relationships and interactions
• User → CLI or Streamlit → invokes CLI dispatcher/frontend controller
• Controller → selects/initializes LlmFactory → produces LLM instances
• Controller/Agent → uses PromptTemplate to format requests → calls LLM.invoke() or LLM.stream()
• Agent → orchestrates calls to Tools (e.g. BrowserTool) and Rankers, retrieving data
• EmbeddingsToolkit → generates embeddings for tools, KB, memory
• VectorDB/KnowledgeBase → store/query embeddings for RAG or long-term memory
• Memory → combines recent chat and long-term via VectorDB → returns context to Agent or Frontend
• Frontend → displays chat history, streams tokens, allows file uploads → updates KnowledgeBase
c. Architectural patterns & design principles
• Modular layered architecture: separation of concerns between UI, orchestration, modeling, data, tools
• Factory pattern: LlmFactory abstracts LLM instantiation/configuration
• Strategy pattern: interchangeable Cores, Embedding toolkits, Rankers, TextSplitters
• Adapter pattern: LangChain interface compatibility
• Plugin-style extensibility: BaseTool, BaseCore, BaseMemory, BaseVectordb
d. Relevant technologies/frameworks
• Python 3.x, PyTorch, llama-cpp-python, ExLlamaV2 (optional)
• FAISS for vector search
• Streamlit for UI
• Pydantic for schemas
• HTTP clients for API calls (OpenAI, HuggingFace, DuckDuckGo)
Putting it all together
1) UI Layer: CLI, Streamlit
2) Orchestration Layer: CLI Dispatcher, Agent, PromptTemplate
3) Core Services Layer:
• LlmFactory → Model Cores
• EmbeddingsToolkit → TextSplitters
• Tools → Rankers → WebSearch, SDTool
4) Data Layer: VectorDB, KnowledgeBase, Memory