# databytes-organisation/discountmate_new architecture

> Architecture diagram of the GitHub repository databytes-organisation/discountmate_new, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databytes-organisation/discountmate_new
- Repository: https://github.com/databytes-organisation/discountmate_new

## Overview

1. **Overall shape**
   - DiscountMate is a multi-repo style monorepo with a production app plus supporting data/ML work.
   - The live user-facing path is: browser/custom domain → Firebase Hosting → Cloud Run frontend → Cloud Run backend → internal ML sidecars.

2. **Frontend subsystem**
   - `Frontend/` is an Expo + React Native Web application built for web delivery.
   - It contains the public shopping experience and the authenticated user hub: product browse/search, categories, product detail, compare, lists, basket, wishlist, profile, notifications, subscription, support, and dashboard pages.
   - The frontend talks to the backend through `EXPO_PUBLIC_API_URL`; it does not connect directly to MongoDB or ML services.

3. **Backend API subsystem**
   - `Backend/` is a Node.js/Express API that acts as the main application boundary.
   - It exposes routes for products, categories, users, baskets, lists, notifications, blogs/news, dashboard, analytics, contact/support, and ML-related features.
   - Swagger is included for API documentation, and middleware covers auth, admin access, throttling, scraping protection, and request validation.

4. **Backend data access boundary**
   - The backend is the only public server-side entrypoint for application data.
   - It connects to MongoDB for app data, user state, and content initialization scripts.
   - This keeps frontend code and external clients isolated from the database and centralizes auth, validation, and business rules.

5. **ML service sidecar**
   - `Backend/ml-service/` is a Python service used for model-backed features such as weekly specials, recommendations, price prediction, OCR receipt processing, and recipe assistance/RAG.
   - In deployment, it runs as a private sidecar container in the same Cloud Run service as the backend and is reached over localhost, not the public internet.
   - The backend proxies user requests to this service, which reduces the public attack surface.

6. **Reverse image search sidecar**
   - `Backend/ml-service/reverse_image_search/` is a separate FastAPI service for product image lookup.
   - It is also deployed as an internal sidecar and loads FAISS assets from Google Cloud Storage.
   - The backend exposes a stable public API route and forwards requests to the local reverse-image-search container.

7. **Data engineering pipeline**
   - `DE/` contains the ingestion and ETL layer that prepares supermarket product data.
   - The `ingestion-pipeline/` scrapes retailer data into local/GCS storage, and `etl-pipeline/` transforms it into structured silver-layer tables and product-history outputs.
   - This pipeline is the source of the application’s curated catalog, pricing, promotions, and product identity normalization.

8. **Scraping and enrichment sources**
   - Several folders under `Scrapping/`, `Catalogue_Scraping_2025/`, and older archived areas contain retailer-specific scrapers, OCR, and catalogue-processing experiments.
   - These are upstream data acquisition tools feeding the DE pipeline and ML experiments rather than the live app directly.
   - They form the ingestion boundary from external retailer websites/catalogues into internal datasets.

9. **ML/analytics experiments vs runtime services**
   - The `ML/`, `Data Analysis/`, and notebook-heavy folders are research and model-development areas.
   - They produce concepts, trained artifacts, and analysis for price prediction, recommendations, reverse image search, OCR, and basket optimisation.
   - Only selected capabilities are operationalized into the runtime services; most notebooks are offline experimentation support.

10. **Infrastructure and deployment**
   - `discount-mate-infra/` defines Terraform for Cloud Run, Artifact Registry, GCS buckets, PostgreSQL, and GitHub Actions identity.
   - The active deployment pattern is containerized services on Google Cloud Run, with images pushed to Artifact Registry.
   - Firebase Hosting provides the custom domain and rewrites all traffic to the frontend Cloud Run service.

11. **CI/CD and release flow**
   - `.github/workflows/` contains the automation for checks and deployment prep.
   - The main workflow validates frontend, backend, and ML code, then builds/pushes images and can deploy to Cloud Run when enabled.
   - This is the release boundary between source control and production cloud infrastructure.

12. **Documentation and operational knowledge**
   - `docs-site/` is the Docusaurus documentation site for architecture, engineering practices, and project handover.
   - The repository also includes many PDFs, notebooks, and archived folders that preserve prior implementations, reports, and alternative approaches.
   - For architecture diagrams, those archived areas should be treated as historical context, not the current production path.

Main entry points: [`Backend`](https://github.com/databytes-organisation/discountmate_new/tree/main/Backend) (Backend API), [`.github/workflows`](https://github.com/databytes-organisation/discountmate_new/blob/main/.github/workflows) (CI/CD), [`docs-site`](https://github.com/databytes-organisation/discountmate_new/tree/main/docs-site) (Docs Site), [`discount-mate-infra`](https://github.com/databytes-organisation/discountmate_new/tree/main/discount-mate-infra) (Infra)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Production App"]
  node_n1(("Browser<br/>client"))
  node_n3["Frontend<br/>expo web app"]
  node_n4["Backend API<br/>node express api"]
  node_n5["API Middleware<br/>auth/guard layer"]
  node_n6["Routes"]
  node_n7["Controllers<br/>request handlers"]
  node_n8[("MongoDB<br/>database")]
  node_n9["ML Proxy<br/>service bridge"]
end

subgraph group_g2["Data Pipeline"]
  node_n12["Ingestion<br/>scraper pipeline"]
  node_n13["ETL<br/>transforms"]
  node_n14["Scrapers<br/>source connectors"]
end

subgraph group_g3["ML Runtime"]
  node_n10["ML Service<br/>python service"]
  node_n11["Reverse Image<br/>fastapi sidecar<br/>[api.py]"]
end

subgraph group_g4["Research & Experiments"]
  node_n15["Research ML<br/>notebooks/models"]
  node_n16["Analytics<br/>analysis workbench"]
end

subgraph group_g5["Delivery & Docs"]
  node_n2["Firebase Hosting<br/>[firebase.json]"]
  node_n17["Infra<br/>terraform cloud"]
  node_n18["CI/CD<br/>github actions<br/>[workflows]"]
  node_n19["Docs Site<br/>docusaurus"]
end

node_n1 -->|"opens"| node_n2
node_n2 -->|"rewrites"| node_n3
node_n3 -->|"API calls"| node_n4
node_n4 -->|"guards"| node_n5
node_n5 -->|"routes"| node_n6
node_n6 -->|"handles"| node_n7
node_n7 -->|"reads/writes"| node_n8
node_n7 -->|"proxies to"| node_n9
node_n9 -->|"invokes"| node_n10
node_n9 -->|"forwards"| node_n11
node_n4 -->|"initializes"| node_n8
node_n12 -->|"feeds"| node_n13
node_n14 -->|"sources"| node_n12
node_n13 -->|"curates"| node_n8
node_n15 -.->|"informs"| node_n10
node_n15 -.->|"informs"| node_n11
node_n16 -.->|"consumes"| node_n13
node_n17 -->|"supports"| node_n2
node_n17 -->|"deploys"| node_n4
node_n17 -->|"deploys"| node_n3
node_n18 -->|"builds"| node_n3
node_n18 -->|"builds"| node_n4
node_n18 -->|"builds"| node_n10
node_n18 -->|"publishes"| node_n19
node_n19 -.->|"documents"| node_n17
node_n19 -.->|"documents"| node_n12
node_n19 -.->|"documents"| node_n15

click node_n2 "https://github.com/databytes-organisation/discountmate_new/blob/main/firebase.json"
click node_n3 "https://github.com/databytes-organisation/discountmate_new/tree/main/Frontend"
click node_n4 "https://github.com/databytes-organisation/discountmate_new/tree/main/Backend"
click node_n5 "https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/src/middleware"
click node_n6 "https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/src/routers"
click node_n7 "https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/src/controllers"
click node_n9 "https://github.com/databytes-organisation/discountmate_new/blob/main/Backend/src/services/reverseImageSearchProcess.js"
click node_n10 "https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/ml-service"
click node_n11 "https://github.com/databytes-organisation/discountmate_new/blob/main/Backend/ml-service/reverse_image_search/api.py"
click node_n12 "https://github.com/databytes-organisation/discountmate_new/tree/main/DE/ingestion-pipeline"
click node_n13 "https://github.com/databytes-organisation/discountmate_new/tree/main/DE/etl-pipeline"
click node_n14 "https://github.com/databytes-organisation/discountmate_new/tree/main/Scrapping"
click node_n15 "https://github.com/databytes-organisation/discountmate_new/tree/main/ML"
click node_n16 "https://github.com/databytes-organisation/discountmate_new/tree/main/Data Analysis"
click node_n17 "https://github.com/databytes-organisation/discountmate_new/tree/main/discount-mate-infra"
click node_n18 "https://github.com/databytes-organisation/discountmate_new/blob/main/.github/workflows"
click node_n19 "https://github.com/databytes-organisation/discountmate_new/tree/main/docs-site"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n3,node_n4,node_n5,node_n6,node_n7,node_n8,node_n9 toneBlue
class node_n12,node_n13,node_n14 toneAmber
class node_n10,node_n11 toneMint
class node_n15,node_n16 toneRose
class node_n2,node_n17,node_n18,node_n19 toneIndigo
```

## Components

### Production App

Live customer-facing product and API path.

- **Browser**: client · Customer entry point for the shopping experience.
- [Frontend](https://github.com/databytes-organisation/discountmate_new/tree/main/Frontend): expo web app · Expo + React Native Web app with shopping and user hub pages. · `Frontend`
- [Backend API](https://github.com/databytes-organisation/discountmate_new/tree/main/Backend): node express api · Main application boundary for product, user, list, dashboard, and ML routes. · `Backend`
- [API Middleware](https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/src/middleware): auth/guard layer · Auth, admin, throttling, anti-scraping, and request validation. · `Backend/src/middleware`
- [Routes](https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/src/routers): api routes · Public route surface for products, categories, baskets, lists, notifications, blog, news, dashboard, analytics, contact, and ML. · `Backend/src/routers`
- [Controllers](https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/src/controllers): request handlers · Business logic for API endpoints and orchestration to services and data. · `Backend/src/controllers`
- **MongoDB**: database · Primary application datastore for user state and content.
- [ML Proxy](https://github.com/databytes-organisation/discountmate_new/blob/main/Backend/src/services/reverseImageSearchProcess.js): service bridge · Backend-side proxy for private ML sidecars and local model services. · `Backend/src/services/reverseImageSearchProcess.js`

### Data Pipeline

Ingestion and ETL that curate catalog and pricing data.

- [Ingestion](https://github.com/databytes-organisation/discountmate_new/tree/main/DE/ingestion-pipeline): scraper pipeline · Retailer scraping and storage layer that collects raw product/catalog data. · `DE/ingestion-pipeline`
- [ETL](https://github.com/databytes-organisation/discountmate_new/tree/main/DE/etl-pipeline): transforms · Feature and mart build layer that normalizes products, pricing, and history. · `DE/etl-pipeline`
- [Scrapers](https://github.com/databytes-organisation/discountmate_new/tree/main/Scrapping): source connectors · Retailer-specific acquisition code feeding the ingestion boundary. · `Scrapping`

### ML Runtime

Operational Python services behind application features.

- [ML Service](https://github.com/databytes-organisation/discountmate_new/tree/main/Backend/ml-service): python service · Private service for specials, recommendations, price prediction, OCR, and recipe assistance. · `Backend/ml-service`
- [Reverse Image](https://github.com/databytes-organisation/discountmate_new/blob/main/Backend/ml-service/reverse_image_search/api.py): fastapi sidecar · Internal reverse-image-search service backed by FAISS assets in cloud storage. · `Backend/ml-service/reverse_image_search/api.py`

### Research & Experiments

Offline notebooks, prototypes, and historical implementations.

- [Research ML](https://github.com/databytes-organisation/discountmate_new/tree/main/ML): notebooks/models · Offline model development for price prediction, recommendations, substitutions, OCR, and search. · `ML`
- [Analytics](https://github.com/databytes-organisation/discountmate_new/tree/HEAD/Data%20Analysis): analysis workbench · Data analysis, dashboards, and reporting artifacts supporting product decisions. · `Data Analysis`

### Delivery & Docs

Infrastructure, CI/CD, hosting, and documentation.

- [Firebase Hosting](https://github.com/databytes-organisation/discountmate_new/blob/main/firebase.json): hosting · Custom domain and rewrite layer in front of the web app. · `firebase.json`
- [Infra](https://github.com/databytes-organisation/discountmate_new/tree/main/discount-mate-infra): terraform cloud · Terraform for Cloud Run, Artifact Registry, storage, databases, and identity. · `discount-mate-infra`
- [CI/CD](https://github.com/databytes-organisation/discountmate_new/blob/main/.github/workflows): github actions · Workflows for checks, release prep, documentation deploy, and service delivery. · `.github/workflows`
- [Docs Site](https://github.com/databytes-organisation/discountmate_new/tree/main/docs-site): docusaurus · Project documentation and handover site for engineering and architecture knowledge. · `docs-site`

## Connections

- Browser → Firebase Hosting: opens · Public traffic enters through the custom domain and hosting layer.
- Firebase Hosting → Frontend: rewrites · Hosting forwards web requests to the frontend service.
- Frontend → Backend API: API calls · Frontend consumes backend APIs via the configured public API URL.
- Backend API → API Middleware: guards · Requests pass through auth, throttling, validation, and anti-scraping middleware.
- API Middleware → Routes: routes · Middleware protects the route surface.
- Routes → Controllers: handles · Routes dispatch to controllers.
- Controllers → MongoDB: reads/writes · Controllers persist and query application state in MongoDB.
- Controllers → ML Proxy: proxies to · Backend forwards ML-related requests to the local service bridge.
- ML Proxy → ML Service: invokes · Proxy reaches the internal ML runtime over localhost/private networking.
- ML Proxy → Reverse Image: forwards · Reverse-image requests are routed to the dedicated sidecar.
- Backend API → MongoDB: initializes · Backend bootstraps content and user data from MongoDB.
- Ingestion → ETL: feeds · Raw ingested data becomes ETL input.
- Scrapers → Ingestion: sources · Retailer scrapers and catalogue tools provide upstream acquisition.
- ETL → MongoDB: curates · Processed product and pricing outputs populate the application datastore boundary.
- Research ML → ML Service: informs · Offline experiments shape runtime ML features and models.
- Research ML → Reverse Image: informs · Reverse-image research informs the sidecar implementation.
- Analytics → ETL: consumes · Analysis and reporting use curated pipeline outputs.
- Infra → Firebase Hosting: supports · Infrastructure defines the hosting and cloud deployment targets.
- Infra → Backend API: deploys · Terraform provisions backend Cloud Run and supporting cloud resources.
- Infra → Frontend: deploys · Infrastructure also supports the frontend deployment path.
- CI/CD → Frontend: builds · CI validates and prepares the frontend application.
- CI/CD → Backend API: builds · CI validates and prepares the backend API.
- CI/CD → ML Service: builds · CI validates and prepares ML service images and checks.
- CI/CD → Docs Site: publishes · Workflow can deploy the documentation site.
- Docs Site → Infra: documents · Docs capture infrastructure and operational knowledge.
- Docs Site → Ingestion: documents · Docs explain the ingestion pipeline architecture.
- Docs Site → Research ML: documents · Docs cover ML and experimentation areas.
