How nigh7wind/gemma4-grocery-agent works
Grocery Agent turns a photo of a handwritten list into a reviewed Instamart cart: the PWA uploads the image, the backend extracts structured items, the user confirms or edits them, and deterministic catalog resolution presents ambiguous choices. The backend then builds and diffs the cart for display and handoff to Swiggy. The graph also preserves the preference store, session state, token handling, and both product-picking interfaces. README documents some capabilities not visible in the excerpts; those links are omitted where their caller is not evidenced.
Main entry points: main.py FastAPI routes · App.jsx App workflow · api.js REST and SSE client · ListConfirmScreen.jsx List review
Components 21
PWA client
- App workflow
frontend/src/App.jsx - REST and SSE client
frontend/src/api.js - List review
frontend/src/components/ListConfirmScreen.jsx - Swipe deck
frontend/src/components/SwipeDeck.jsx - Swipe card
frontend/src/components/SwipeCard.jsx - List item card
frontend/src/components/ListItemCard.jsx - Product swipe picker
frontend/src/components/ProductDecisionScreen.jsx - Product grid picker
frontend/src/components/ProductGridScreen.jsx - Product card
frontend/src/components/ProductCard.jsx
Backend service
- FastAPI routes
app/main.py - Checkout session state
app/sessions.py - Data contracts
app/schemas.py - List extraction
app/vision.py
Catalog matching
- Product resolution
app/resolution.py - Preference memory
app/preferences.py
External systems
- Swiggy MCP client
app/mcp_client.py - OAuth token store
app/token_store.py
Other components
- Shopper
- Local vision LLM
- Swiggy Instamart
- Swiggy app
Connections 26
- Shopper → App workflow: uses
README.md - App workflow → REST and SSE client: calls
frontend/src/App.jsx - List review → REST and SSE client: edits list
frontend/src/components/ListConfirmScreen.jsx - Product swipe picker → REST and SSE client: submits choices
frontend/src/components/ProductDecisionScreen.jsx - Product grid picker → REST and SSE client: submits choices
frontend/src/components/ProductGridScreen.jsx - List review → Swipe deck: renders
frontend/src/components/ListConfirmScreen.jsx - Swipe deck → Swipe card: renders
frontend/src/components/SwipeDeck.jsx - List review → List item card: renders
frontend/src/components/ListConfirmScreen.jsx - Product swipe picker → Swipe deck: renders
frontend/src/components/ProductDecisionScreen.jsx - Product swipe picker → Product card: renders
frontend/src/components/ProductDecisionScreen.jsx - App workflow → List review: shows
frontend/src/App.jsx - App workflow → Product swipe picker: shows
frontend/src/App.jsx - App workflow → Product grid picker: shows
frontend/src/App.jsx - REST and SSE client → FastAPI routes: HTTP and SSE
README.md - FastAPI routes → Checkout session state: reads and updates
app/main.py - FastAPI routes → List extraction: extracts photo
app/main.py - FastAPI routes → Product resolution: resolves items
app/main.py - FastAPI routes → Swiggy MCP client: builds cart
app/main.py - FastAPI routes → OAuth token store: uses token state
app/main.py - FastAPI routes → Data contracts: uses requests
app/main.py - List extraction → Local vision LLM: invokes
app/vision.py - Product resolution → Preference memory: checks preferences
app/resolution.py - Product resolution → Swiggy MCP client: searches inventory
app/resolution.py - Swiggy MCP client → Swiggy Instamart: calls MCP tools
app/mcp_client.py - FastAPI routes → Swiggy app: hands off cart
README.md - Swiggy Instamart → Swiggy app: opens checkout
README.md