How luisvilriv/comprobador-de-paginas-web works
1. This repo is a containerized web-auditing platform split into three runtime pieces: a browser frontend, a FastAPI dashboard backend, and a separate Python scraper/auditor worker. The shared code under shared/ is the main boundary that keeps the API and worker consistent.
2. The user-facing entry point is docker/dashboard/frontend/, served by docker/dashboard/server.js. The frontend is a modular JavaScript app (api.js, websites.js, scheduler.js, audit.js, modals.js, i18n.js) that talks only to the dashboard API, not directly to the database or scraper.
3. The dashboard backend lives in docker/dashboard/api/ and is organized by domain routes: clients, websites, runs, settings, summary, plus health. This API is the orchestration layer for the UI: it exposes CRUD and reporting endpoints, validates payloads with Pydantic schemas, and delegates persistence to the shared database repositories.
4. The scraper side is a separate execution service under scraper/ and docker/scraper/. It is structured as a pipeline: a context selects a scraping strategy, a service layer coordinates the run, and the result is stored through shared repository code. The Docker entrypoints (entrypoint.py, scheduler.py, service.py) indicate it can run either on demand or as a scheduled worker.
5. Scraping is strategy-based. scraper/strategies/beautifulsoup_strategy.py and selenium_strategy.py represent two crawl modes, with base/ defining the abstraction and context/ selecting the implementation. This separation is important because the architecture supports both lightweight HTML parsing and browser-driven execution.
6. The audit engine is centralized in shared/auditor/. It is not tied to the scraper implementation; instead, it consumes scraped page data and runs categorized checks in checks/ such as browser, buttons, content, images, links, security, SEO, structure, and technical. Scoring and rule dictionaries live alongside the checks, so the audit logic is reusable across services.
7. Persistence is also shared. shared/database/connection.py and shared/database/models.py define the database layer, while repository packages are split by consumer: repositories/dashboard/ for API-facing queries and repositories/scraper/ for worker writes/reads. This keeps DB access behind a repository facade and prevents the dashboard and scraper from depending on raw SQLAlchemy internals.
8. The database is PostgreSQL, initialized by docker/db-init/01_schema.sql and optionally migrated by 02_migrate_optional_client.sql. That means schema creation happens as part of the container startup flow, not through application bootstrapping.
9. There is an optional AI analysis service under docker/ai-analyzer/, with its own app, analyzer, and requirements. The presence of dedicated tests (test_ai_analyzer_*) suggests it enriches or post-processes audit results rather than replacing the core crawler/auditor pipeline.
10. Configuration is centralized in config/settings.py and logging in config/logging_config.py. The repo also relies heavily on environment-driven deployment via docker/.env.example, which matches the container-first design.
11. Report/export support exists but is secondary to the main flow. shared/utils/pdf_generator.py suggests the system can generate downloadable artifacts from audit data, likely consumed by the dashboard summary/report views.
12. The dominant data flow is: frontend request -> dashboard API -> shared repositories -> PostgreSQL for management data, and scheduler/worker run -> scrape pages -> audit checks -> persist results -> dashboard reads summaries/runs back from the same database. That shared database is the integration point between all major subsystems.
13. The test suite mirrors the architecture: endpoint tests for the API, unit tests for individual audit checks, Selenium/strategy tests for scraping behavior, and integration tests for the AI analyzer. This is a good sign that the repo is organized around boundaries that are meant to be independently verifiable.
Main entry points: routes Domain Routes · service.py Run Service · auditor Auditor Core · config Config
Components 18
Frontend
Browser UI for dashboard users.
- Dashboard UI · User-facing modular JavaScript frontend.
docker/dashboard/frontend - Server · Serves the dashboard frontend and connects browser requests to the API.
docker/dashboard/server.js
Dashboard API
FastAPI orchestration layer for client, website, run, and summary operations.
- API App · Dashboard backend application entrypoint.
docker/dashboard/api/app.py - Domain Routes · Client, website, run, settings, summary, and health endpoints.
docker/dashboard/api/routes - Schemas · Request and response validation models for dashboard endpoints.
docker/dashboard/api/schemas
Scraper Worker
Separate scraping and auditing execution service.
- Worker Entry · Scraper container entrypoints for scheduled or on-demand runs.
docker/scraper - Run Service · Coordinates scraping execution and persistence.
docker/scraper/service.py - Scraper Context · Chooses the active scraping strategy.
scraper/context/scraper_context.py - Strategies · BeautifulSoup and Selenium scraping implementations behind a common abstraction.
scraper/strategies
Shared Core
Common audit, database, and utility code reused by services.
- Auditor Core · Runs reusable audit logic over scraped page data.
shared/auditor - Checks · Categorized audit checks for browser, content, links, security, SEO, structure, and technical issues.
shared/auditor/checks - DB Layer · Shared database connection and models.
shared/database - Dashboard Repos · Repository facade used by the API for dashboard-facing queries and mutations.
shared/database/repositories/dashboard - Scraper Repos · Repository facade used by the worker to persist scrape and audit results.
shared/database/repositories/scraper - PDF Export · Generates downloadable report artifacts from audit data.
shared/utils/pdf_generator.py
Infrastructure
Container and database bootstrap pieces.
- PostgreSQL · Database initialized by container SQL scripts.
docker/db-init/01_schema.sql - AI Analyzer · Optional analysis service that enriches or post-processes audit results.
docker/ai-analyzer - Config · Centralized settings and logging configuration.
config
Connections 19
- Server → Dashboard UI: serves · Node server delivers the browser UI.
- Dashboard UI → API App: calls · Frontend communicates only with the dashboard API.
- API App → Domain Routes: mounts · FastAPI app exposes domain route modules.
- Domain Routes → Schemas: validates · Endpoints use schemas for payload validation.
- Domain Routes → Dashboard Repos: reads/writes · API delegates persistence to dashboard repositories.
- Dashboard Repos → DB Layer: uses · Dashboard repositories use shared DB models and connection logic.
- Worker Entry → Run Service: starts · Container entrypoints launch the worker service.
- Run Service → Scraper Context: selects · Service chooses the active scraping strategy.
- Scraper Context → Strategies: dispatches · Context delegates to BeautifulSoup or Selenium implementations.
- Run Service → Auditor Core: audits · Scraped page data is passed into the shared auditor.
- Auditor Core → Checks: runs · Audit engine executes categorized rule sets.
- Auditor Core → Scraper Repos: persists · Audit and scrape results are stored through scraper repositories.
- Scraper Repos → DB Layer: uses · Worker repositories share the same database layer.
- DB Layer → PostgreSQL: backed by · Database layer targets the PostgreSQL schema initialized in containers.
- Domain Routes → PDF Export: reports · Summary endpoints can generate report artifacts.
- AI Analyzer → Scraper Repos: consumes · Optional analyzer works from stored scrape/audit data.
- Config → API App: configures · Settings influence API runtime behavior.
- Config → Run Service: configures · Settings influence worker runtime behavior.
- Config → Auditor Core: configures · Shared settings and logging shape audit execution.