How harshaltupe12/screenshare works
Project Type and Purpose
• This is a full-stack, real-time web application (“Jerry – AI-Powered Screen Assistant”) that lets users share their browser screen, runs OCR on the frames, and returns contextual AI responses (text and voice) via WebSockets.
• Frontend: Next.js 14 + TypeScript, React components, MediaDevices API, canvas capture, react-speech-kit.
• Backend: Node.js + Express + Socket.IO, MongoDB, JWT auth, Tesseract OCR, Gemini/Hugging Face AI integrations.
File Structure Analysis
• Top level: backend/ and frontend/.
• backend/src/: MVC-style split into controllers, routes, services, models, middleware, utils, plus config files and test scripts.
• frontend/app/ (Next.js App Router pages), frontend/components/ (UI and domain components), frontend/lib/ (API, auth, WebSocket helpers), plus config (tsconfig, ESLint, postcss, next.config).
Key README Insights
• Real-time features: WebSocket for screen frames and AI responses.
• Authentication: JWT with refresh, password hashing, account lockout.
• AI pipeline: screen capture → base64 → WebSocket → OCR → AI API → response → persistence → WebSocket back.
• Data storage: MongoDB Atlas, four main collections (User, Meeting, Message, AISession).
Diagramming the System
a. Main Components
• Frontend Browser (Next.js App)
• Backend API Server (Express + Socket.IO)
• Database (MongoDB Atlas)
• OCR Engine (Tesseract via eng.traineddata)
• AI Providers (Gemini AI, Hugging Face)
• Auth subsystem (JWT, bcrypt)
• External Services: WebSocket clients, TTS/ASR in browser
b. Relationships & Data Flows
1. User → Frontend UI (React pages/components)
2. Frontend REST → Backend Auth routes → issue JWT
3. User triggers “Share Screen” → MediaDevices API + Canvas → base64 images
4. Frontend WebSocket → Backend /ai-session route
5. Backend Service chain:
• OCR Service extracts text
• AI Service calls external AI provider
• Persist to MongoDB (models: AISession, Message)
6. Backend WebSocket → Frontend → display chat + play TTS
7. Frontend Speech Recognition → text input → loop
c. Architectural Patterns & Principles
• Separation of concerns (MVC on backend, feature-based directories on frontend).
• Event-driven, real-time communication (Socket.IO).
• Stateless JWT authentication with middleware guards.
• Service layer abstracts AI, OCR, TTS.
• Environment-based configuration (aiConfig.js, database.js).
d. Key Technologies/Frameworks
• Frontend: Next.js 14, React, TypeScript, react-speech-kit, WebSocket API, MediaDevices
• Backend: Node.js, Express, Socket.IO, Mongoose, Tesseract, Axios (for AI APIs), bcrypt, jsonwebtoken
• Database: MongoDB Atlas
• Deployment: environment variables, cross-origin setup (FRONTEND_URL)
Essential Details to Capture
• All four backend controllers and their corresponding routes (auth, ai, aiSession, meetings).
• Service dependencies: aiController → aiService → external AI; aiSessionController → ocrService → aiService → ttsService.
• JWT flow: client sends token → auth middleware validates → requests processed.
• WebSocket handshake path (/socket.io or custom namespace).