How ashrith915/offline-llm-content-generator works
a. User Interface (Frontend):
• Represented by the Streamlit web interface.
• This component captures user inputs, displays generated content, and shows interactive visualizations.
b. AI Content Generation Engine (Backend):
• This core component is embodied by the “optimized-llm.py” script.
• It handles the integration of the LLaMA model files for text generation.
• GPU acceleration via PyTorch with CUDA support is used here.
c. Speech-to-Text Module:
• Use of Vosk for speech recognition.
• This module processes audio/speech input and converts it to text.
d. Visualization and Data Processing:
• This includes components that utilize Pandas and Plotly to generate interactive visualizations of the content metrics.
e. External Models & Configurations:
• LLaMA and Vosk model files stored locally.
• Presence of different model files for low VRAM modes (4GB GPU compatibility) and higher quality options.
f. Dependency and Build System:
• Briefly note the existence of configuration files like requirements-file.txt, which lists Python dependencies required for the project.
Architectural Patterns and Design Principles
• Separation of Concerns: Clearly distinguish between frontend (Streamlit UI), processing (content generation and speech-to-text), and output (visualizations).
• Modular Design: Even though the project is a single-file driven app, conceptually separate the “modules” within the code that handle different concerns.
• Offline Functioning: Emphasize that all models and services (like Vosk and LLaMA) are run locally, with no dependency on online APIs.
Technology and Frameworks to Annotate
• Python 3.8+ – the language of choice.
• Streamlit – for the frontend web interface.
• PyTorch with CUDA – for GPU-based model processing.
• Vosk – for speech recognition.
• LangChain and CTransformers – for parts of the language model handling.
• Pandas and Plotly – for creating visualizations.