# pawan8010/tender_project_app_updated architecture

> Architecture diagram of the GitHub repository pawan8010/tender_project_app_updated, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pawan8010/tender_project_app_updated
- Repository: https://github.com/pawan8010/tender_project_app_updated

## Overview

1. **What this repo is**
   - Apna Tender is a tender aggregation platform, not just a scraper. It ingests procurement notices from Indian government portals, normalizes them, stores them, matches them against keywords/ML rules, and exposes the results in a dashboard.

2. **Top-level runtime shape**
   - The system is split into a **React/Vite frontend** and a **FastAPI backend**.
   - In local mode, the backend can run its own background scraping loop.
   - In Docker/production mode, scraping is split into **Celery worker + Celery Beat + Redis**, with **PostgreSQL** as the primary database.

3. **Frontend boundary**
   - `frontend/src` is a client UI built with React 18 and Vite.
   - It talks to the backend through the API layer in `frontend/src/lib/api.js`.
   - The UI is organized around dashboard-like pages such as Tenders, Keywords, Alerts, Sessions, and System, with components for filtering, search, export, language switching, and live status display.
   - A key architectural point: the frontend is presentation-only; it does not own business logic or scraping.

4. **Backend application boundary**
   - `backend/app/main.py` is the FastAPI entrypoint.
   - The backend is divided into routers, models, schemas, auth/config/database helpers, and services.
   - Routers under `backend/app/routers` provide the public API surface for health, auth, tenders, keywords, scrape control, logs, alerts, backups, export, statistics, users, workers, and related dashboard features.
   - There is also an SSE/realtime scrape stream endpoint, which is used for live status updates rather than polling.

5. **Data model and persistence**
   - SQLAlchemy models in `backend/app/models.py` define the persisted domain.
   - The database abstraction in `backend/app/database.py` supports SQLite for local runs and PostgreSQL for Docker/production.
   - Core persisted entities, based on the README and file layout, include tenders, tender matches, portal runs, scrape logs, users/auth state, and operational records for backup/status/history.
   - Startup initializes the schema automatically; seeds are controlled by environment flags.

6. **Scraping subsystem**
   - The scraper layer lives in `backend/scrapers`.
   - `registry.py` and `portal_manager.py` act as the control plane for which portals are enabled and how they are run.
   - `base_scraper.py` provides shared parsing behavior, while `portals/` contains portal-specific implementations for sites like CPPP, GeM, GePNIC, IREPS, Karnataka, Gujarat, Andhra Pradesh, Telangana, Bihar, and others.
   - The scrapers use a mix of HTTP fetching and HTML parsing, with **Playwright** reserved for portals that require browser rendering or JavaScript execution.

7. **Scrape execution and orchestration**
   - A scrape run starts from the API or scheduler, then moves into the worker/orchestrator layer.
   - `backend/scheduler/orchestrator.py`, `backend/scrapers/worker.py`, and `backend/celery_worker.py` coordinate background execution depending on runtime mode.
   - Concurrency, retries, portal timeouts, and run logging are all managed on the backend side, not in the UI.
   - This is the main operational boundary in the system: portal access is isolated from the request/response API so scraping can continue independently of user traffic.

8. **Normalization, upsert, and change detection**
   - Scraped rows are not stored raw only; they are normalized and then upserted into the database.
   - Services such as `change_detector.py`, `document_processor.py`, `tender_index.py`, `text_quality.py`, and `summaries.py` support post-scrape processing.
   - The architecture expects change detection to compare incoming tender data against existing records, so updates can be logged and surfaced as deltas rather than just inserts.
   - This is important for the dashboard because “what changed” is a first-class concept.

9. **Search, keyword matching, and ML enrichment**
   - Keyword logic is implemented in `backend/app/services/keyword_engine.py` and exposed through keyword-related routers and keyword suggestion endpoints.
   - `ml_engine.py`, `ai_intelligence.py`, and `ml_search.py` provide optional semantic/local AI ranking or matching.
   - The system can fall back to local fuzzy/keyword matching if ML dependencies are unavailable.
   - There is also a tender index layer, which suggests the app supports searching across normalized tender records rather than only browsing portal-specific data.

10. **Discovery and external search**
   - `google_tender_discovery.py` integrates Google Custom Search as an optional discovery path.
   - This is a separate enrichment/discovery source, not the primary scraper pipeline.
   - Its role is to find portal pages or external tender-like results that can then be normalized and stored alongside portal-scraped data.

11. **Alerts, notifications, and operational outputs**
   - `backend/app/services/notification.py` and `backend/app/routers/alerts.py` handle email-style alerting when tenders match configured criteria.
   - `backend/app/services/export.py` and the export router generate CSV/Excel outputs, which is reflected in the checked-in `exports/` artifacts.
   - Backup snapshots are also a built-in concern, with `backend/app/services/backup.py` and a `backups/` directory indicating filesystem-based retention of tender snapshots or matched records.

12. **Observability and operational endpoints**
   - Health, logs, dashboard, statistics, and workers routes show that operational visibility is part of the architecture, not an afterthought.
   - `backend/app/routers/health.py` and the portal health/log endpoints are used to show scrape success, database status, and recent portal activity.
   - The UI likely depends on these endpoints for system pages and live monitoring.

13. **Auth and access control**
   - Authentication is handled in `backend/app/auth.py` and exposed through the auth router.
   - The app uses JWT-based access tokens and seeded admin/user configuration from environment variables.
   - This makes the dashboard a secured operational console rather than a public search page.

14. **Deployment and infrastructure**
   - Docker Compose is the production-style topology: backend, worker, scheduler/beat, PostgreSQL, Redis, and frontend served via Nginx.
   - The repo also includes scripts for local Windows startup and a preview server for serving the built frontend.
   - CI runs backend compile checks, pytest, and frontend build validation, so the architecture is validated at both API and UI layers.

15. **Main end-to-end data flow**
   - Portal content enters through scrapers or Google discovery, is normalized and upserted through the backend service layer, stored in SQLAlchemy-backed tables, then surfaced through REST/SSE to the React UI.
   - On the way, it may be enriched by keyword matching, ML ranking, document processing, backups, exports, and notification workflows.
   - The key architectural separation is: **scraping and enrichment are backend worker concerns; display, filtering, and user actions are frontend concerns**.

Main entry points: [`backend/app/routers`](https://github.com/pawan8010/tender_project_app_updated/tree/main/backend/app/routers) (API Routers), [`frontend/src/App.jsx`](https://github.com/pawan8010/tender_project_app_updated/blob/main/frontend/src/App.jsx) (App UI), [`backend/scheduler/orchestrator.py`](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scheduler/orchestrator.py) (Orchestrator), [`backend/app/services/change_detector.py`](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/change_detector.py) (Change & Match)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_frontend_app["App UI<br/>React/Vite app<br/>[App.jsx]"]
  node_frontend_pages["Dashboard Pages<br/>UI pages"]
  node_frontend_components["UI Components<br/>presentation components"]
  node_frontend_api["API Client<br/>HTTP client<br/>[api.js]"]
end

subgraph group_backend_api["Backend API"]
  node_backend_main["FastAPI App<br/>API entrypoint<br/>[main.py]"]
  node_backend_routers["API Routers<br/>REST surface"]
  node_backend_models[("Data Models<br/>SQLAlchemy models<br/>[models.py]")]
  node_backend_db[("Database Layer<br/>DB config<br/>[database.py]")]
  node_backend_auth["Auth<br/>[auth.py]"]
end

subgraph group_scraping["Scraping"]
  node_scrape_orchestrator["Orchestrator<br/>background coordination<br/>[orchestrator.py]"]
  node_scrape_worker["Worker<br/>scrape runtime<br/>[worker.py]"]
  node_celery_worker["Celery Worker<br/>async worker<br/>[celery_worker.py]"]
  node_portal_manager["Portal Manager<br/>control plane<br/>[portal_manager.py]"]
  node_scraper_base["Base Scraper<br/>shared scraper logic<br/>[base_scraper.py]"]
  node_portal_scrapers["Portal Scrapers<br/>portal implementations"]
end

subgraph group_services["Services"]
  node_change_match["Change & Match<br/>processing services<br/>[change_detector.py]"]
  node_search_enrichment["Search & AI<br/>enrichment services<br/>[tender_index.py]"]
  node_outputs["Outputs<br/>export/notify services<br/>[export.py]"]
end

subgraph group_infra["Infrastructure"]
  node_runtime["Deployment<br/>docker/runtime<br/>[docker-compose.yml]"]
end

node_frontend_app -->|"renders"| node_frontend_pages
node_frontend_app -->|"uses"| node_frontend_components
node_frontend_app -->|"calls"| node_frontend_api
node_frontend_pages -->|"composes"| node_frontend_components
node_frontend_api -->|"requests"| node_backend_routers
node_backend_main -->|"mounts"| node_backend_routers
node_backend_routers -->|"auth"| node_backend_auth
node_backend_routers -->|"reads/writes"| node_backend_models
node_backend_models -->|"persists via"| node_backend_db
node_backend_routers -->|"starts runs"| node_scrape_orchestrator
node_scrape_orchestrator -->|"dispatches"| node_scrape_worker
node_scrape_orchestrator -->|"queues"| node_celery_worker
node_scrape_worker -->|"coordinates"| node_portal_manager
node_celery_worker -->|"coordinates"| node_portal_manager
node_portal_manager -->|"extends"| node_scraper_base
node_scraper_base -->|"specializes"| node_portal_scrapers
node_portal_scrapers -->|"feeds"| node_change_match
node_change_match -->|"enriches"| node_search_enrichment
node_change_match -->|"triggers"| node_outputs
node_backend_routers -->|"exposes"| node_outputs
node_backend_main -->|"deploys in"| node_runtime
node_frontend_app -->|"served by"| node_runtime

click node_frontend_app "https://github.com/pawan8010/tender_project_app_updated/blob/main/frontend/src/App.jsx"
click node_frontend_pages "https://github.com/pawan8010/tender_project_app_updated/tree/main/frontend/src/pages"
click node_frontend_components "https://github.com/pawan8010/tender_project_app_updated/tree/main/frontend/src/components"
click node_frontend_api "https://github.com/pawan8010/tender_project_app_updated/blob/main/frontend/src/lib/api.js"
click node_backend_main "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/main.py"
click node_backend_routers "https://github.com/pawan8010/tender_project_app_updated/tree/main/backend/app/routers"
click node_backend_models "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/models.py"
click node_backend_db "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/database.py"
click node_backend_auth "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/auth.py"
click node_scrape_orchestrator "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scheduler/orchestrator.py"
click node_scrape_worker "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scrapers/worker.py"
click node_celery_worker "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/celery_worker.py"
click node_portal_manager "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scrapers/portal_manager.py"
click node_scraper_base "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scrapers/base_scraper.py"
click node_portal_scrapers "https://github.com/pawan8010/tender_project_app_updated/tree/main/backend/scrapers/portals"
click node_change_match "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/change_detector.py"
click node_search_enrichment "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/tender_index.py"
click node_outputs "https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/export.py"
click node_runtime "https://github.com/pawan8010/tender_project_app_updated/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_app,node_frontend_pages,node_frontend_components,node_frontend_api toneBlue
class node_backend_main,node_backend_routers,node_backend_models,node_backend_db,node_backend_auth toneAmber
class node_scrape_orchestrator,node_scrape_worker,node_celery_worker,node_portal_manager,node_scraper_base,node_portal_scrapers toneMint
class node_change_match,node_search_enrichment,node_outputs toneRose
class node_runtime toneIndigo
```

## Components

### Frontend

React/Vite presentation layer

- [App UI](https://github.com/pawan8010/tender_project_app_updated/blob/main/frontend/src/App.jsx): React/Vite app · Client-side dashboard shell and page routing · `frontend/src/App.jsx`
- [Dashboard Pages](https://github.com/pawan8010/tender_project_app_updated/tree/main/frontend/src/pages): UI pages · Tender, keyword, alert, session, and system views · `frontend/src/pages`
- [UI Components](https://github.com/pawan8010/tender_project_app_updated/tree/main/frontend/src/components): presentation components · Shared filters, tables, search, export, and status widgets · `frontend/src/components`
- [API Client](https://github.com/pawan8010/tender_project_app_updated/blob/main/frontend/src/lib/api.js): HTTP client · Frontend integration point for backend requests · `frontend/src/lib/api.js`

### Backend API

FastAPI app, routers, models, auth, and persistence

- [FastAPI App](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/main.py): API entrypoint · Backend application bootstrap and router wiring · `backend/app/main.py`
- [API Routers](https://github.com/pawan8010/tender_project_app_updated/tree/main/backend/app/routers): REST surface · Health, auth, tenders, keywords, scrape control, logs, alerts, export, statistics, users, workers · `backend/app/routers`
- [Data Models](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/models.py): SQLAlchemy models · Persisted tenders, matches, runs, logs, users, and operational records · `backend/app/models.py`
- [Database Layer](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/database.py): DB config · SQLite and PostgreSQL connection and session management · `backend/app/database.py`
- [Auth](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/auth.py): JWT auth · Login, tokens, and access control helpers · `backend/app/auth.py`

### Scraping

Portal crawling and background orchestration

- [Orchestrator](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scheduler/orchestrator.py): background coordination · Schedules and coordinates scrape runs across runtime modes · `backend/scheduler/orchestrator.py`
- [Worker](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scrapers/worker.py): scrape runtime · Background execution for local or distributed scraping · `backend/scrapers/worker.py`
- [Celery Worker](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/celery_worker.py): async worker · Production scrape execution process · `backend/celery_worker.py`
- [Portal Manager](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scrapers/portal_manager.py): control plane · Enables, selects, and runs portal scrapers · `backend/scrapers/portal_manager.py`
- [Base Scraper](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/scrapers/base_scraper.py): shared scraper logic · Common parsing and fetch behavior for portals · `backend/scrapers/base_scraper.py`
- [Portal Scrapers](https://github.com/pawan8010/tender_project_app_updated/tree/main/backend/scrapers/portals): portal implementations · CPPP, GeM, GePNIC, IREPS, state portals, and NIC variants · `backend/scrapers/portals`

### Services

Normalization, matching, enrichment, and outputs

- [Change & Match](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/change_detector.py): processing services · Upsert, change detection, keyword matching, text quality, and summaries · `backend/app/services/change_detector.py`
- [Search & AI](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/tender_index.py): enrichment services · Tender index, ML search, semantic scoring, and discovery · `backend/app/services/tender_index.py`
- [Outputs](https://github.com/pawan8010/tender_project_app_updated/blob/main/backend/app/services/export.py): export/notify services · Alerts, notifications, exports, backups, and document processing · `backend/app/services/export.py`

### Infrastructure

Docker, deployment, and runtime support

- [Deployment](https://github.com/pawan8010/tender_project_app_updated/blob/main/docker-compose.yml): docker/runtime · Frontend, backend, worker, scheduler, Redis, and PostgreSQL topology · `docker-compose.yml`

## Connections

- App UI → Dashboard Pages: renders · App composes dashboard views
- App UI → UI Components: uses · App reuses shared UI widgets
- App UI → API Client: calls · UI talks to backend through API client
- Dashboard Pages → UI Components: composes · Pages assemble reusable controls and tables
- API Client → API Routers: requests · Frontend consumes REST/SSE endpoints
- FastAPI App → API Routers: mounts · FastAPI entrypoint wires routes
- API Routers → Auth: auth · Auth routes and token handling
- API Routers → Data Models: reads/writes · API operates on persisted domain data
- Data Models → Database Layer: persists via · ORM models use the database layer
- API Routers → Orchestrator: starts runs · Scrape control endpoints trigger orchestration
- Orchestrator → Worker: dispatches · Local mode execution path
- Orchestrator → Celery Worker: queues · Distributed production execution path
- Worker → Portal Manager: coordinates · Worker delegates portal selection and execution
- Celery Worker → Portal Manager: coordinates · Celery runtime delegates portal selection and execution
- Portal Manager → Base Scraper: extends · Portal implementations share base scraping behavior
- Base Scraper → Portal Scrapers: specializes · Portal-specific scrapers implement concrete sources
- Portal Scrapers → Change & Match: feeds · Scraped records are normalized, compared, and upserted
- Change & Match → Search & AI: enriches · Change detection and indexing support search and ranking
- Change & Match → Outputs: triggers · Matches can produce alerts, exports, and backups
- API Routers → Outputs: exposes · Export, alerts, documents, and statistics endpoints
- FastAPI App → Deployment: deploys in · App is packaged in the production topology
- App UI → Deployment: served by · Built UI is served through the deployment stack
