How bnbong/qshing_server works
Project type and purpose
• This is a backend phishing-detection microservice (“Qshing Server”) built in Python 3.12 using FastAPI.
• It exposes a POST API endpoint (/phishing-detection/analyze) that accepts a URL, crawls the page (via a local WebDriver), extracts text and images, and runs a multimodal CNN+BERT model to predict phishing.
File structure
• Top-level:
• Dockerfile, docker-compose.yml, build-spec.yaml (containerization & deployment)
• alembic.ini + migrations/ (database schema migrations)
• scripts/ (db init, lint, format, test, dependency fetch)
• src/qshing_server/: the application code
• Under src/qshing_server:
• api/: FastAPI routers & dependency definitions
• core/: config loader, custom exceptions
• dto/: Pydantic request/response schemas
• service/: business logic
– parser/: HTML loader via WebDriver
– model/: preprocessor, tokenizer, Q-BERT model, model manager (loads checkpoint .pt)
– phishing_analyzer.py: orchestrates parsing + model inference
• db/: db_manager.py (connects to Mongo or SQL), ORM models
• utils/: logging, enums
• main.py: FastAPI app instantiation, middleware, router inclusion
Readme insights
• Emphasizes Docker usage (local vs. container runtime differences when crawling).
• Notes AI model file must be placed in service/model.
• Mentions environment variables (SECRET_KEY, MODEL_NAME).
• External Client (curl/Postman/browser)
• API Layer: Uvicorn + FastAPI server
• Router: phishing_routers
• Dependencies (injection of config, DB session)
• Service Layer
• Parser subcomponent: HTML loader (uses local Chrome WebDriver)
• Analyzer: coordinates feature extraction + inference
• Model Layer
• Preprocessor → Tokenizer → Q-BERT (CNN+BERT) modules
• Model Manager (loads checkpoint file)
• Data Layer
• Database (MongoDB or SQL via Alembic migrations)
• db_manager & ORM models for storing request logs or stats
• Infrastructure
• Docker container(s)
• Docker Compose / CI workflows
• External WebDriver dependency
b. Relationships & interactions
1. Client → HTTP POST → FastAPI router
2. Router → inject config & DB session → hands off URL to PhishingAnalyzer
3. Analyzer → Parser loads HTML via WebDriver → returns raw data
4. Analyzer → Preprocessor transforms data → Tokenizer encodes text → Q-BERT infers → returns result + confidence
5. Analyzer → Optionally store request & result to DB via db_manager
6. FastAPI → Pydantic DTO constructs response → returns JSON to Client
c. Architectural patterns & design principles
• Layered Architecture: clear separation between API, service logic, model inference, and persistence.
• Dependency Injection via FastAPI Depends for config and DB sessions.
• Single-responsibility modules (parser, analyzer, model manager).
• Containerized deployment (Docker + Compose) for environment parity.
d. Key technologies/frameworks
• FastAPI + Pydantic for REST API
• Uvicorn as ASGI server
• PyTorch (or TensorFlow) for CNN+BERT inference
• Alembic for migrations
• Docker & Docker Compose for containerization
• Selenium/Chromedriver (implied) for HTML crawling
• CI via GitHub Actions