How bharathsr16/ai_smartclassroom works
Project type and purpose
• This is a standalone, multimodal AI assistant (not a full‐stack web app) that runs locally, orchestrated by a central Python script. Its purpose is to interact with students via text, voice, and vision, answer questions, transcribe lectures, summarize, gauge engagement, generate visual aids, and log feedback.
File structure
Top-level grouping:
• Main orchestrator: main_assistant.py, app_ui.py, summarize_lecture_cli.py
• Text modules: text_qa_module.py, text_summarizer.py, text_summarizer_ov.py, convert_summarizer_to_ov.py
• Voice modules: voice_processing_module.py, lecture_transcriber.py
• Visual modules: visual_module.py, visual_aid_generator.py
• Engagement & feedback: engagement_module.py, feedback_reporter.py, feedback_log.csv
• Model conversion & IR storage: convert_bert.py, bert_ir/, ov_model_placeholder/
• Utilities & data: feedback_log.csv, placeholder model files, XML IR files
Extract key insights from README
• Central orchestrator integrates all modules but currently fails in sandbox.
• Each module depends on specific external libraries (PyTorch/Transformers, OpenCV, SpeechRecognition/PyAudio, pyttsx3, fer/TensorFlow, Matplotlib).
• OpenVINO conversion path exists for optimization.
• Hardware dependencies: mic, speaker, webcam, TTS engine, Haar cascade XML.
1. User Interface
• CLI (summarize_lecture_cli.py)
• Simple GUI (app_ui.py)
2. Orchestrator
• main_assistant.py
• Text QA & Summarization
• Voice (STT & TTS)
• Visual Processing & Captioning
• Engagement Monitoring
• Visual Aid Generation
4. Model Layer
• Hugging Face Transformers Models
• OpenVINO IR Models (bert_ir/, ov_model_placeholder/)
5. Utilities & Data Stores
• Feedback Reporter & CSV Log
6. External Dependencies & Hardware
• Microphone, Speaker, Webcam
• OS-level engines (eSpeak-NG, PortAudio, Tesseract OCR)
• Haarcascade XML
• User interacts with UI → UI calls Orchestrator
• Orchestrator routes requests to the appropriate module → module loads model (HF or OpenVINO) → processes input → returns result to Orchestrator → Orchestrator outputs via UI
• Engagement module continuously pulls camera feed → analyzes → streams engagement metrics back to Orchestrator
• Feedback Reporter listens for specific events/responses → appends to CSV store
c. Architectural patterns & design principles
• Modular “plugin” pattern: each modality is a self-contained module with a well-defined interface
• Mediator pattern: main_assistant.py acts as the central coordinator
• Separation of concerns: UI, orchestration, AI logic, model storage, and reporting are distinct layers
• Extensibility via conversion scripts for OpenVINO optimization
d. Key technologies & frameworks
• Python 3.9+, Hugging Face Transformers + PyTorch
• OpenVINO (IR model directory)
• OpenCV for vision & face detection
• speech_recognition + PyAudio, pyttsx3 for voice
• fer (TensorFlow) for emotion recognition
• Matplotlib for chart generation
• Tesseract OCR (optional advanced visual queries)