How aswad546/sso-monitor works
<explanation>
1. Project type and purpose
- This is a full‑stack, microservices‑style web application (“SSO‑Monitor”) for continuously scanning millions of web sites’ login pages to detect and archive Single‑Sign‑On support.
- The “brain” directory is a Flask web front‑end + API; “worker” and “worker‑backend” are background compute services (crawler, analyzers, LLM classification); supporting services include MongoDB, MinIO, RabbitMQ, Prometheus, Promtail, searxng, Jupyter, and a cookie‑store microservice.
2. Top‑level directories as architectural clues
• brain/
– Flask app.py, blueprints for API and views, static assets, Jinja2 templates, config modules (MongoDB, RabbitMQ, MinIO, Celery, logging).
• worker/
– CLI‐driven Python service that pulls tasks, orchestrates login‐page detection (using worker/modules/*), writes results to MongoDB/MinIO via the API.
• worker‑backend/
– GPU‐accelerated classification backends (vllm serve for OS‑Atlas, Qwen2.5‑VL).
• watcher/
– lightweight Python service monitoring long‑running tasks/workers.
• cookie‑store/
– microservice for hosted cookie storage (Pipfile + script).
• searxng/, prometheus/, promtail/, rabbitmq/, jupyter/
– each an external service container for search aggregation, metrics collection, log shipping, message broker, notebook environment.
3. Key insights from README
- The system is launched via Docker‑Compose.
- Ports mapped: web UI (8080), DB UI (8081/9090), RabbitMQ (5672), MongoDB (27017), MinIO, Redis, Jupyter (8888), Prometheus (9090).
- Workflow: user triggers scan via web UI → brain pushes a Celery/RabbitMQ task → worker(s) fetch tasks, run analyzers, call external/searxng crawler, image classifiers (hosted externally via vllm serve), push results back → brain reads from MongoDB + MinIO and presents.
4. How to draw the system design diagram
a. Main components (boxes):
- Client/browser
- Reverse proxy (Traefik or Docker‑Compose network)
- brain (Flask web + API)
- RabbitMQ (task broker)
- Celery workers:
• worker (login‑page detection, analyzers)
• worker‑backend (LLM classifier & screenshot classifier)
• watcher
- searxng search service
- Cookie‑store service
- MongoDB (document store)
- MinIO (object store)
- Monitoring: Prometheus + Promtail
- Jupyter notebook
b. Relationships & data flows (arrows):
- User → HTTP → traefik → brain web UI
- brain API → RabbitMQ (enqueue task)
- worker/worker‑backend/watcher → RabbitMQ (fetch tasks)
- worker → searxng (REST) for discovery → HTTP server at worker/modules/loginpagedetection (http.server:8001) → vllm services (gRPC/REST)
- All workers → MongoDB (results) + MinIO (large files)
- brain → reads MongoDB + MinIO → displays results
- Prometheus scrapes metrics endpoints on brain, workers, searxng
- Promtail collects service logs
- Jupyter connects to MongoDB/MinIO for ad‑hoc analysis
c. Architectural patterns
- Microservices communicating via message queue (RabbitMQ + Celery).
- MVC pattern in brain (blueprints for API vs views).
- External “model serving” microservices (vllm) for classification.
- Service discovery within Docker‑Compose network.
- Observability stack (Prometheus + promtail).
d. Technologies & frameworks
- Flask (brain), Celery (task queue), RabbitMQ (broker), MongoDB (data), MinIO (storage), vllm serve (LLM), Python modules for analyzers, searxng (search), Jupyter, Prometheus, Promtail, Docker‑Compose.
5. Tailoring the diagram
- Emphasize front‑end vs API separation in brain:
– Box “Flask web UI” for templates/static assets; box “Flask API” for blueprints under /api.
- Show RabbitMQ as the hand‑off between brain and workers.
- Worker‑backend microservices for GPU‑heavy tasks should be separate boxes, labeled with model names (OS‑Atlas, Qwen2.5).
- Open‑source “searxng” shows integration point for site discovery.
- Cookie‑store as a small sidecar service for storing identity provider cookies.
- Monitoring stack as separate “observability” group.
6. Diagram elements to include
• Clear component labels (e.g., “brain: Flask Web + API (port 8080)”)
• Directional arrows for HTTP calls, message enqueue/dequeue, DB writes/reads
• Color coding or shapes:
– Blue for front‑end/API
– Orange for message queue + task bus
– Green for background workers
– Purple for data stores
– Gray for external/third‑party services
• Annotate ports and protocols where important (e.g., RabbitMQ 5672, HTTP 8001/8002, MongoDB 27017)
7. Detail and completeness
- Don’t lump all background work into one box – separate analyzers, LLM classification services, crawler, watcher.
- Show data artifacts vs metadata: MinIO vs MongoDB.
- Capture observability (Prometheus scrape targets, promtail log paths).
- Include Jupyter as a user‑facing analysis interface.
- Represent Docker‑Compose network as a container boundary if you like, but show intra‑service connections clearly.
By following this breakdown you will produce a comprehensive, accurate architecture diagram that captures SSO‑Monitor’s front‑to‑back workflow, modular services, and key infrastructure elements.