How nicholasching/perception works
Main Components
• Mobile Frontend: At the heart of the project is the React Native mobile interface. The file structure under the "app" folder—with separate components for camera, settings, index, and not-found pages—indicates a tab-based navigation structure.
• Utility Module: The "app/utils/gemini_util.tsx" points to an abstraction handling image recognition logic (likely interfacing with Gemini 2.0 Flash Lite).
• Device-Specific Backend Functionality: Although no traditional backend exists in the project repository, the README describes local technologies integrated into the app:
• Speech Transcription: Utilizes native services (SFSpeechRecognizer for iOS and SpeechRecognizer for Android).
• Image Recognition: Uses Gemini 2.0 Flash Lite, managed through utility functions.
• External Libraries/Services: Incorporate the use of Expo (including expo-speech for text-to-speech) which interfaces directly into the mobile app’s functionality.
Relationships and Data Flow
• Camera Flow: Show flow from capturing an image in the camera tab → passing the image to the Gemini utility function for analysis → retrieving a detected description.
• Voice Feedback: After obtaining a description, illustrate how the app calls the text-to-speech (expo-speech) module to convert text to audible speech.
• Integration of Utility Logic: Illustrate how the gemini_util.tsx module interfaces between the captured image and the Gemini 2.0 Flash Lite service.
Architectural Patterns and Technologies
• Separation of Concerns: Emphasize the logical separation between the user interface (React Native components and navigation layouts) and core business logic (utility functions, integrations with native features).
• Layered Architecture:
• Presentation Layer (UI/UX screens, tabs, layouts)
• Business Logic Layer (image recognition utility, voice command handlers)
• Native Integration Layer (expo-speech, SFSpeechRecognizer, and Android SpeechRecognizer)
a. "Mobile App" container, labeling it as running on React Native with Expo.
b. Inside it, create a branch for the UI components representing each tab:
• Index/Home Screen
• Settings Screen
• Not-found handling (for invalid routes)
• Gemini_util.tsx as the bridge to external image recognition (Gemini 2.0 Flash Lite).
d. Integrations with native APIs:
e. Insert the expo-speech component as a service that converts processed textual descriptions to spoken words.
f. Optionally, include a “Navigation/Layout” block to show how app/(tabs)/_layout.tsx coordinates the UI flow.
Final Reminders for the Principal Software Engineer
• That all components are clearly delineated with labels.