How aswad546/visiblev8-crawler works
Project Type and Purpose
• This is a distributed, full‑stack crawling framework built around the VisibleV8 engine.
• It provides a CLI for users to submit URLs, a Python/FastAPI backend that schedules crawling jobs via Celery, two distinct Celery worker services (log parser and VV8 crawler), and persisting of results in PostgreSQL (metadata) and MongoDB (screenshots, HARs, logs).
• The project uses Docker and Docker Compose to orchestrate all services, and includes observability stacks (Flower UI, Prometheus, Loki, Promtail, OpenTelemetry) for monitoring tasks and infrastructure.
File‑Structure Analysis
• Top‑level directories:
• backend – FastAPI service, Alembic migrations, Celery configuration, unit tests
• celery_workers – two sub‑services, each with its own Dockerfile, tasks, configuration, and tests
• scripts – CLI entrypoint (vv8-cli.py), helper utilities, local SQLite store
• Infrastructure manifests – docker‑compose.yaml, compose for builds, PostgreSQL init, MongoDB init, Prometheus, Loki, OTEL‑collector, promtail
• Key patterns:
• API & Task queue separation: FastAPI enqueues jobs to Celery broker (e.g., Redis/RabbitMQ)
• Micro‑services: distinct Docker images for each worker role
• Database migrations via Alembic
• Monitoring via standard observability tools
README Insights
• Describes CLI commands (setup, crawl, fetch) that talk to the backend or directly query databases/Flower UI.
• Highlights the two data stores (PostgreSQL for structured data, MongoDB for unstructured artifacts)
• References Flower UI on port 5555 for Celery task tracking
• CLI Client (vv8-cli.py)
• API Service (backend FastAPI)
• Celery Broker (Redis or RabbitMQ)
• Celery Workers – log_parser_worker and vv8_worker
• VisibleV8 Engine (invoked inside vv8_worker container)
• Postgres Database (vv8_backend_database)
• MongoDB (for HARs, logs, screenshots)
• Monitoring & Observability
– Flower UI
– Prometheus + Prometheus.yml
– Loki + Promtail
– OpenTelemetry Collector
1. CLI Client → submits HTTP request → API Service
2. API Service → publishes task message → Celery Broker
3. Celery Broker → dispatches tasks → Workers
4. vv8_worker → runs VisibleV8 binary → generates artifacts → writes to MongoDB
5. vv8_worker → emits metadata (e.g., timings, status) → writes to PostgreSQL
6. log_parser_worker → consumes raw logs → post‑processes → updates PostgreSQL
7. API Service → reads from PostgreSQL (status, metadata) → returns to CLI
8. Monitoring:
• Flower UI ↔ Celery Broker/Workers (HTTP)
• Prometheus scrapes metrics endpoints (API, Workers, OTEL)
• Loki & Promtail collect logs from containers
• OTEL Collector centralizes traces/metrics
c. Architectural Patterns & Principles:
• Event‑driven, asynchronous task processing via Celery
• Clear separation of concerns: CLI, API, worker roles
• Infrastructure as code with Docker Compose
• Database migrations for schema evolution
• Observability baked in (metrics, logs, traces)
d. Key Technologies & Frameworks:
• Python 3.10+, FastAPI, Uvicorn
• Celery with Redis/RabbitMQ broker
• PostgreSQL (with Alembic)
• MongoDB
• Docker, Docker Compose
• Flower, Prometheus, Loki, Promtail, OpenTelemetry Collector
Essential Elements Checklist
• [ ] Labels with technology names and ports
• [ ] Cluster boundaries for logical grouping