How pawan8010/tender_project_app_updated works
1. What this repo is
- Apna Tender is a tender aggregation platform, not just a scraper. It ingests procurement notices from Indian government portals, normalizes them, stores them, matches them against keywords/ML rules, and exposes the results in a dashboard.
2. Top-level runtime shape
- The system is split into a React/Vite frontend and a FastAPI backend.
- In local mode, the backend can run its own background scraping loop.
- In Docker/production mode, scraping is split into Celery worker + Celery Beat + Redis, with PostgreSQL as the primary database.
3. Frontend boundary
- frontend/src is a client UI built with React 18 and Vite.
- It talks to the backend through the API layer in frontend/src/lib/api.js.
- The UI is organized around dashboard-like pages such as Tenders, Keywords, Alerts, Sessions, and System, with components for filtering, search, export, language switching, and live status display.
- A key architectural point: the frontend is presentation-only; it does not own business logic or scraping.
4. Backend application boundary
- backend/app/main.py is the FastAPI entrypoint.
- The backend is divided into routers, models, schemas, auth/config/database helpers, and services.
- Routers under backend/app/routers provide the public API surface for health, auth, tenders, keywords, scrape control, logs, alerts, backups, export, statistics, users, workers, and related dashboard features.
- There is also an SSE/realtime scrape stream endpoint, which is used for live status updates rather than polling.
5. Data model and persistence
- SQLAlchemy models in backend/app/models.py define the persisted domain.
- The database abstraction in backend/app/database.py supports SQLite for local runs and PostgreSQL for Docker/production.
- Core persisted entities, based on the README and file layout, include tenders, tender matches, portal runs, scrape logs, users/auth state, and operational records for backup/status/history.
- Startup initializes the schema automatically; seeds are controlled by environment flags.
6. Scraping subsystem
- The scraper layer lives in backend/scrapers.
- registry.py and portal_manager.py act as the control plane for which portals are enabled and how they are run.
- base_scraper.py provides shared parsing behavior, while portals/ contains portal-specific implementations for sites like CPPP, GeM, GePNIC, IREPS, Karnataka, Gujarat, Andhra Pradesh, Telangana, Bihar, and others.
- The scrapers use a mix of HTTP fetching and HTML parsing, with Playwright reserved for portals that require browser rendering or JavaScript execution.
7. Scrape execution and orchestration
- A scrape run starts from the API or scheduler, then moves into the worker/orchestrator layer.
- backend/scheduler/orchestrator.py, backend/scrapers/worker.py, and backend/celery_worker.py coordinate background execution depending on runtime mode.
- Concurrency, retries, portal timeouts, and run logging are all managed on the backend side, not in the UI.
- This is the main operational boundary in the system: portal access is isolated from the request/response API so scraping can continue independently of user traffic.
8. Normalization, upsert, and change detection
- Scraped rows are not stored raw only; they are normalized and then upserted into the database.
- Services such as change_detector.py, document_processor.py, tender_index.py, text_quality.py, and summaries.py support post-scrape processing.
- The architecture expects change detection to compare incoming tender data against existing records, so updates can be logged and surfaced as deltas rather than just inserts.
- This is important for the dashboard because “what changed” is a first-class concept.
9. Search, keyword matching, and ML enrichment
- Keyword logic is implemented in backend/app/services/keyword_engine.py and exposed through keyword-related routers and keyword suggestion endpoints.
- ml_engine.py, ai_intelligence.py, and ml_search.py provide optional semantic/local AI ranking or matching.
- The system can fall back to local fuzzy/keyword matching if ML dependencies are unavailable.
- There is also a tender index layer, which suggests the app supports searching across normalized tender records rather than only browsing portal-specific data.
10. Discovery and external search
- google_tender_discovery.py integrates Google Custom Search as an optional discovery path.
- This is a separate enrichment/discovery source, not the primary scraper pipeline.
- Its role is to find portal pages or external tender-like results that can then be normalized and stored alongside portal-scraped data.
11. Alerts, notifications, and operational outputs
- backend/app/services/notification.py and backend/app/routers/alerts.py handle email-style alerting when tenders match configured criteria.
- backend/app/services/export.py and the export router generate CSV/Excel outputs, which is reflected in the checked-in exports/ artifacts.
- Backup snapshots are also a built-in concern, with backend/app/services/backup.py and a backups/ directory indicating filesystem-based retention of tender snapshots or matched records.
12. Observability and operational endpoints
- Health, logs, dashboard, statistics, and workers routes show that operational visibility is part of the architecture, not an afterthought.
- backend/app/routers/health.py and the portal health/log endpoints are used to show scrape success, database status, and recent portal activity.
- The UI likely depends on these endpoints for system pages and live monitoring.
13. Auth and access control
- Authentication is handled in backend/app/auth.py and exposed through the auth router.
- The app uses JWT-based access tokens and seeded admin/user configuration from environment variables.
- This makes the dashboard a secured operational console rather than a public search page.
14. Deployment and infrastructure
- Docker Compose is the production-style topology: backend, worker, scheduler/beat, PostgreSQL, Redis, and frontend served via Nginx.
- The repo also includes scripts for local Windows startup and a preview server for serving the built frontend.
- CI runs backend compile checks, pytest, and frontend build validation, so the architecture is validated at both API and UI layers.
15. Main end-to-end data flow
- Portal content enters through scrapers or Google discovery, is normalized and upserted through the backend service layer, stored in SQLAlchemy-backed tables, then surfaced through REST/SSE to the React UI.
- On the way, it may be enriched by keyword matching, ML ranking, document processing, backups, exports, and notification workflows.
- The key architectural separation is: scraping and enrichment are backend worker concerns; display, filtering, and user actions are frontend concerns.
Main entry points: routers API Routers · App.jsx App UI · orchestrator.py Orchestrator · change_detector.py Change & Match
Components 19
Frontend
React/Vite presentation layer
- App UI · Client-side dashboard shell and page routing
frontend/src/App.jsx - Dashboard Pages · Tender, keyword, alert, session, and system views
frontend/src/pages - UI Components · Shared filters, tables, search, export, and status widgets
frontend/src/components - API Client · Frontend integration point for backend requests
frontend/src/lib/api.js
Backend API
FastAPI app, routers, models, auth, and persistence
- FastAPI App · Backend application bootstrap and router wiring
backend/app/main.py - API Routers · Health, auth, tenders, keywords, scrape control, logs, alerts, export, statistics, users, workers
backend/app/routers - Data Models · Persisted tenders, matches, runs, logs, users, and operational records
backend/app/models.py - Database Layer · SQLite and PostgreSQL connection and session management
backend/app/database.py - Auth · Login, tokens, and access control helpers
backend/app/auth.py
Scraping
Portal crawling and background orchestration
- Orchestrator · Schedules and coordinates scrape runs across runtime modes
backend/scheduler/orchestrator.py - Worker · Background execution for local or distributed scraping
backend/scrapers/worker.py - Celery Worker · Production scrape execution process
backend/celery_worker.py - Portal Manager · Enables, selects, and runs portal scrapers
backend/scrapers/portal_manager.py - Base Scraper · Common parsing and fetch behavior for portals
backend/scrapers/base_scraper.py - Portal Scrapers · CPPP, GeM, GePNIC, IREPS, state portals, and NIC variants
backend/scrapers/portals
Services
Normalization, matching, enrichment, and outputs
- Change & Match · Upsert, change detection, keyword matching, text quality, and summaries
backend/app/services/change_detector.py - Search & AI · Tender index, ML search, semantic scoring, and discovery
backend/app/services/tender_index.py - Outputs · Alerts, notifications, exports, backups, and document processing
backend/app/services/export.py
Infrastructure
Docker, deployment, and runtime support
- Deployment · Frontend, backend, worker, scheduler, Redis, and PostgreSQL topology
docker-compose.yml
Connections 22
- App UI → Dashboard Pages: renders · App composes dashboard views
- App UI → UI Components: uses · App reuses shared UI widgets
- App UI → API Client: calls · UI talks to backend through API client
- Dashboard Pages → UI Components: composes · Pages assemble reusable controls and tables
- API Client → API Routers: requests · Frontend consumes REST/SSE endpoints
- FastAPI App → API Routers: mounts · FastAPI entrypoint wires routes
- API Routers → Auth: auth · Auth routes and token handling
- API Routers → Data Models: reads/writes · API operates on persisted domain data
- Data Models → Database Layer: persists via · ORM models use the database layer
- API Routers → Orchestrator: starts runs · Scrape control endpoints trigger orchestration
- Orchestrator → Worker: dispatches · Local mode execution path
- Orchestrator → Celery Worker: queues · Distributed production execution path
- Worker → Portal Manager: coordinates · Worker delegates portal selection and execution
- Celery Worker → Portal Manager: coordinates · Celery runtime delegates portal selection and execution
- Portal Manager → Base Scraper: extends · Portal implementations share base scraping behavior
- Base Scraper → Portal Scrapers: specializes · Portal-specific scrapers implement concrete sources
- Portal Scrapers → Change & Match: feeds · Scraped records are normalized, compared, and upserted
- Change & Match → Search & AI: enriches · Change detection and indexing support search and ranking
- Change & Match → Outputs: triggers · Matches can produce alerts, exports, and backups
- API Routers → Outputs: exposes · Export, alerts, documents, and statistics endpoints
- FastAPI App → Deployment: deploys in · App is packaged in the production topology
- App UI → Deployment: served by · Built UI is served through the deployment stack