How lokeshkumar3103ar/memoji works
1. Overall shape
- This is a single-page browser app with a small serverless API layer for anything that needs secrets or external lookups.
- The primary runtime is the browser; the backend is just a couple of Vercel API routes under api/.
2. Frontend shell
- index.html, styles.css, and app.js form the UI layer.
- index.html likely bootstraps the camera/meme experience, while styles.css handles the visual layout for the webcam view, overlays, and controls.
- app.js is the main orchestrator: it handles UI state, camera access, face/emotion detection, meme/emoji selection, capture/share actions, and calling backend endpoints.
3. Real-time computer vision pipeline
- The app uses face-api.js in the browser, as called out in the README.
- The flow is: camera stream → face detection → emotion classification → UI update.
- The detected expression is the core signal that drives all downstream behavior.
4. Emotion-to-reaction mapping
- The client maintains local mappings from emotions to:
- emoji reactions
- meme search queries
- The README explicitly mentions editable CONFIG.EMOTION_QUERIES and CONFIG.EMOTION_EMOJI in app.js, so app.js is not just behavior code; it also contains the app’s main content/configuration tables.
5. Meme mode vs emoji overlay mode
- The UI appears to support at least two distinct presentation modes:
- Emoji overlay mode: replaces or overlays the face with a large emoji.
- Meme match mode: fetches a relevant GIF/meme based on the detected emotion.
- This mode switch is an important product boundary because it changes whether the app is purely local rendering or involves a network call.
6. Serverless API boundary
- The api/ directory contains api/analyze.js and api/meme.js, which are deployed as serverless functions on Vercel.
- This is the architecture’s trust boundary: the browser should not hold secrets, so anything involving AI or third-party API keys should be routed through these functions.
7. External service integration
- The README lists Azure OpenAI environment variables and an optional Tenor API key.
- That means the app has at least two external-service dependencies:
- Azure OpenAI for analysis/AI-assisted logic
- Tenor for meme/GIF retrieval
- These are kept behind the API routes so the browser never needs to know the credentials.
8. Capture and media assets
- assets/faaah.mp3 and assets/vine-boom-sound-meme.mp3 indicate the app includes audio reactions tied to certain states/events.
- This is a local asset subsystem: no network dependency, just browser playback when an emotion/meme event fires.
- The app likely also captures still images from the video stream for download/share.
9. Deployment/runtime model
- vercel.json defines how the project is hosted on Vercel.
- The deployment model is a static frontend plus auto-routed API functions, which fits the README’s “secure API architecture” note.
- The app is otherwise host-agnostic and can also run from a simple static server for local testing.
10. Configuration and setup files
- config.example.js and generate_config.js suggest local/environment-specific config generation or templating.
- These are supporting files, not runtime architecture pillars, but they indicate the app expects configurable endpoints/keys during setup.
- MEMOJI APP AUDIT.md is documentation/audit material, likely useful for maintenance rather than the runtime flow.
11. Data flow summary
- The main end-to-end path is:
- user opens page
- browser requests camera permission
- webcam frames are processed locally
- emotion is inferred
- UI updates with emoji/audio
- if meme mode is active, a serverless API call returns a matching GIF or AI-derived result
- The key architectural idea is local-first perception with server-side secret handling only where needed.
12. Important diagram boundaries
- Draw a hard boundary around the browser/client versus Vercel serverless/API.
- Draw another boundary around external services: Azure OpenAI and Tenor.
- Treat the webcam and face detection as a client-only real-time processing loop, not as backend processing.
Main entry points: app.js Main App · analyze.js Analyze API · meme.js Meme API · index.html Page Shell
Components 18
Browser App
Static frontend and real-time client-side interaction loop.
- Page Shell · Entry document that loads the app UI and bootstraps the browser experience.
index.html - App Styles · Visual styling for the webcam view, overlays, and controls.
styles.css - Main App · Coordinates camera access, face/emotion detection, reaction state, capture/share actions, and API calls.
app.js - Camera Loop · Browser-side webcam stream and frame processing loop.
- Face API.js · In-browser face detection and expression classification library used by the app.
- Reaction Map · Local emotion-to-emoji and emotion-to-query tables maintained in the client.
app.js - Emoji Mode · Presentation mode that overlays or replaces the face with a large emoji.
- Meme Mode · Presentation mode that requests a matching meme or GIF for the detected emotion.
- Capture Share · Still capture and sharing/download actions triggered from the UI.
API Layer
Vercel serverless functions used for secret-backed lookups.
- Analyze API · Serverless analysis endpoint that keeps secret-backed AI logic off the client.
api/analyze.js - Meme API · Serverless endpoint that resolves a suitable meme or GIF for the current emotion.
api/meme.js
External Services
Third-party services reached through the API boundary.
- Azure OpenAI · External AI service used behind the analyze endpoint.
- Tenor · External GIF/meme source used behind the meme endpoint.
Media Assets
Local audio reactions bundled with the app.
- Faaah Sound · Local audio reaction asset bundled with the app.
assets/faaah.mp3 - Vine Boom · Local audio reaction asset bundled with the app.
assets/vine-boom-sound-meme.mp3
Other components
- Vercel Config · Hosting and function routing configuration for Vercel.
vercel.json - Config Template · Example configuration used to generate or document local environment settings.
config.example.js - Config Generator · Helper for generating environment-specific configuration.
generate_config.js
Connections 21
- Page Shell → App Styles: loads · Page shell includes the stylesheet.
- Page Shell → Main App: boots · Page shell loads the main browser orchestrator.
- Main App → Camera Loop: controls · Main app starts and manages the webcam processing loop.
- Camera Loop → Face API.js: detects · Webcam frames are analyzed with face-api.js.
- Face API.js → Main App: expressions · Detected expressions feed back into application state.
- Main App → Reaction Map: uses · Client logic looks up emoji and query mappings for the current emotion.
- Reaction Map → Emoji Mode: drives · Emotion mapping selects the emoji reaction.
- Reaction Map → Meme Mode: drives · Emotion mapping selects the meme search/query path.
- Main App → Emoji Mode: renders · Main app updates the local emoji overlay presentation.
- Main App → Meme Mode: requests · Main app requests meme content when meme mode is active.
- Meme Mode → Meme API: fetches · Browser calls the meme endpoint for a matching GIF or meme.
- Main App → Analyze API: analyzes · Client can call the analysis endpoint for secret-backed AI logic.
- Analyze API → Azure OpenAI: uses · Analyze function delegates to Azure OpenAI.
- Meme API → Tenor: queries · Meme function queries Tenor for reactions.
- Main App → Capture Share: handles · Main app manages capture and share/download actions.
- Main App → Faaah Sound: plays · App can trigger bundled audio reactions.
- Main App → Vine Boom: plays · App can trigger bundled audio reactions.
- Vercel Config → Analyze API: routes · Deployment config exposes the analyze function.
- Vercel Config → Meme API: routes · Deployment config exposes the meme function.
- Config Generator → Config Template: derives · Generator uses the example config as a template.
- Config Template → Main App: configures · Client behavior references generated runtime settings.