How sign/translate works
Project type and purpose
• This is a full-stack, real-time sign-language translation application (“Sign Translate”) with:
• A rich Angular/Ionic front-end for desktop and mobile (wrapped via Capacitor)
• A serverless back-end on Firebase Functions exposing HTTP APIs
• TFJS and Mediapipe models loaded in the client for on-device inference
• GitHub Actions CI/CD, Fastlane pipelines for mobile publishing, and VitePress docs
File-structure analysis
• Top-level “src/” folder holds the Angular app with:
• “app/” – components, modules, pages (Translation pages, Landing, Settings, Playground)
• “environments/” – environment.ts (prod/test/dev)
• “main.ts”, “server.ts”, “main.server.ts” – hybrid client/server entrypoints
• “functions/” – Firebase Functions code with:
• “gateway/” – HTTP controllers (avatars, me, spoken-to-signed, etc.)
• “text-normalization/”, “text-to-text/”, “prerender/”, plus shared middlewares
• Mobile wrappers: “android/” and “ios/” with Capacitor/Gradle and Xcode projects, plus Fastlane metadata
• “docs/” – VitePress documentation site
• CI/CD: “.github/workflows” for client, server, Lighthouse audits
README insights
• It highlights two core flows:
a. Spoken-to-Signed (Audio → Text → Normalization → SignWriting → Pose → Skeleton/3D Avatar → Human GAN)
b. Signed-to-Spoken (Video Upload/Camera → Segmentation → SignWriting → Text → Audio)
• The front-end uses modules (pose, detector, pix2pix, sign-writing, translate) with NGXS state management
• TFJS models and web workers are used for on-device inference
Key components & interactions
a. Frontend (Angular/Ionic)
• UI pages/components for capture (video, audio), model loading (TFJS, Mediapipe), translation display
• Services: HTTP client → Firebase Functions endpoints; TFJS loader/service; Google Analytics; AppCheck
• State: NGXS stores for video frames, translation state, pose, settings
• Workers: pix2pix.worker.ts for heavy ML tasks off the main thread
b. Back-end (Firebase Functions)
• Gateway routers expose REST endpoints: /avatars, /spoken-to-signed, /me, /prerender, /text-normalization, /text-to-text
• Middlewares: AppCheck validation, rate-limiting, error handling
• Controllers orchestrate model invocation or proxy to model servers / TFJS on Cloud Functions
• (Optionally) Cloud Storage holds large model files, accessed by client and functions
c. External services
• Google AppCheck for bot protection
• Google Analytics for usage tracking
• Firebase Hosting / Cloud Functions for deployment
• Fastlane for mobile app distribution
d. Local assets
• /assets/models – TFJS and Mediapipe model files loaded by the client
• /assets/promotional – media for marketing and docs site
Architectural patterns & design principles
• Client-heavy: offload inference to browser via TFJS, Mediapipe, and Web Workers
• Reactive state management (NGXS) guiding UI flows
• Serverless micro-services: each function folder is an independent HTTP endpoint
• Clear separation: front-end vs. back-end vs. mobile wrapper vs. docs
Don’t overthink—be exhaustive
• Break the front-end into UI, state, ML services, HTTP client
• Break the back-end into gateway, controllers, model logic, middlewares
• CI/CD pipelines and documentation site as side components