# aswad546/visiblev8-crawler architecture

> Architecture diagram of the GitHub repository aswad546/visiblev8-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aswad546/visiblev8-crawler
- Repository: https://github.com/aswad546/visiblev8-crawler

## Overview

##### Project Type and Purpose
- This is a distributed, full‑stack crawling framework built around the VisibleV8 engine.
- It provides a CLI for users to submit URLs, a Python/FastAPI backend that schedules crawling jobs via Celery, two distinct Celery worker services (log parser and VV8 crawler), and persisting of results in PostgreSQL (metadata) and MongoDB (screenshots, HARs, logs).
- The project uses Docker and Docker Compose to orchestrate all services, and includes observability stacks (Flower UI, Prometheus, Loki, Promtail, OpenTelemetry) for monitoring tasks and infrastructure.

##### File‑Structure Analysis
- Top‑level directories:
- backend – FastAPI service, Alembic migrations, Celery configuration, unit tests
- celery_workers – two sub‑services, each with its own Dockerfile, tasks, configuration, and tests
- scripts – CLI entrypoint (vv8-cli.py), helper utilities, local SQLite store
- Infrastructure manifests – docker‑compose.yaml, compose for builds, PostgreSQL init, MongoDB init, Prometheus, Loki, OTEL‑collector, promtail
- Key patterns:
- API & Task queue separation: FastAPI enqueues jobs to Celery broker (e.g., Redis/RabbitMQ)
- Micro‑services: distinct Docker images for each worker role
- Database migrations via Alembic
- Monitoring via standard observability tools

##### README Insights
- Describes CLI commands (setup, crawl, fetch) that talk to the backend or directly query databases/Flower UI.
- Highlights the two data stores (PostgreSQL for structured data, MongoDB for unstructured artifacts)
- References Flower UI on port 5555 for Celery task tracking

- CLI Client (vv8-cli.py)
- API Service (backend FastAPI)
- Celery Broker (Redis or RabbitMQ)
- Celery Workers – log_parser_worker and vv8_worker
- VisibleV8 Engine (invoked inside vv8_worker container)
- Postgres Database (vv8_backend_database)
- MongoDB (for HARs, logs, screenshots)
- Monitoring & Observability
– Flower UI
– Prometheus + Prometheus.yml
– Loki + Promtail
– OpenTelemetry Collector
1. CLI Client → submits HTTP request → API Service
2. API Service → publishes task message → Celery Broker
3. Celery Broker → dispatches tasks → Workers
4. vv8_worker → runs VisibleV8 binary → generates artifacts → writes to MongoDB
5. vv8_worker → emits metadata (e.g., timings, status) → writes to PostgreSQL
6. log_parser_worker → consumes raw logs → post‑processes → updates PostgreSQL
7. API Service → reads from PostgreSQL (status, metadata) → returns to CLI
8. Monitoring:
- Flower UI ↔ Celery Broker/Workers (HTTP)
- Prometheus scrapes metrics endpoints (API, Workers, OTEL)
- Loki & Promtail collect logs from containers
- OTEL Collector centralizes traces/metrics
c. Architectural Patterns & Principles:
- Event‑driven, asynchronous task processing via Celery
- Clear separation of concerns: CLI, API, worker roles
- Infrastructure as code with Docker Compose
- Database migrations for schema evolution
- Observability baked in (metrics, logs, traces)
d. Key Technologies & Frameworks:
- Python 3.10+, FastAPI, Uvicorn
- Celery with Redis/RabbitMQ broker
- PostgreSQL (with Alembic)
- MongoDB
- Docker, Docker Compose
- Flower, Prometheus, Loki, Promtail, OpenTelemetry Collector

##### Essential Elements Checklist
- [ ] Labels with technology names and ports
- [ ] Cluster boundaries for logical grouping

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Services
    CLI["CLI Client\nscripts/vv8-cli.py"]:::service
    API["API Service\nbackend/app/main.py"]:::service
    Broker["Celery Broker\n(Redis/RabbitMQ)"]:::broker

    subgraph "Crawler Cluster"
        logWorker["Log Parser Worker\ncelery_workers/log_parser_worker"]:::service
        vv8Worker["VV8 Crawler Worker\ncelery_workers/vv8_worker"]:::service
        VisibleV8["VisibleV8 Engine\nvv8_crawler"]:::service
    end

    %% Datastores
    Postgres["PostgreSQL\nvv8_backend_database"]:::datastore
    Mongo["MongoDB\nmongo/init"]:::datastore

    %% Monitoring & Observability
    subgraph "Monitoring & Observability"
        Flower["Flower UI\nflower"]:::monitor
        Prometheus["Prometheus"]:::monitor
        Loki["Loki"]:::monitor
        Promtail["Promtail"]:::monitor
        OTEL["OpenTelemetry Collector"]:::monitor
    end

    %% Flows
    CLI -->|"HTTP request"| API
    API -->|"enqueue task"| Broker
    Broker -->|"dispatch task"| logWorker
    Broker -->|"dispatch task"| vv8Worker
    vv8Worker -->|"run engine"| VisibleV8
    VisibleV8 -->|"write artifacts"| Mongo
    vv8Worker -->|"write metadata"| Postgres
    logWorker -->|"update metadata"| Postgres
    API -->|"read metadata"| Postgres

    %% Monitoring flows
    Flower <-->|"monitor tasks"| Broker
    Prometheus -->|"scrape metrics"| API
    Prometheus -->|"scrape metrics"| vv8Worker
    Prometheus -->|"scrape metrics"| logWorker
    Prometheus -->|"scrape metrics"| OTEL
    Promtail -->|"collect logs"| API
    Promtail -->|"collect logs"| vv8Worker
    Promtail -->|"collect logs"| logWorker
    OTEL -->|"collect traces/metrics"| API
    OTEL -->|"collect traces/metrics"| vv8Worker
    OTEL -->|"collect traces/metrics"| logWorker

    %% Click Events
    click CLI "https://github.com/aswad546/visiblev8-crawler/blob/main/scripts/vv8-cli.py"
    click API "https://github.com/aswad546/visiblev8-crawler/blob/main/backend/app/main.py"
    click Broker "https://github.com/aswad546/visiblev8-crawler/blob/main/docker-compose.yaml"
    click logWorker "https://github.com/aswad546/visiblev8-crawler/blob/main/celery_workers/log_parser_worker/app.py"
    click vv8Worker "https://github.com/aswad546/visiblev8-crawler/blob/main/celery_workers/vv8_worker/app.py"
    click VisibleV8 "https://github.com/aswad546/visiblev8-crawler/tree/main/celery_workers/vv8_worker/vv8_crawler/"
    click Postgres "https://github.com/aswad546/visiblev8-crawler/tree/main/vv8_backend_database/Dockerfile"
    click Mongo "https://github.com/aswad546/visiblev8-crawler/blob/main/mongo/init/mongo-init.js"
    click Flower "https://github.com/aswad546/visiblev8-crawler/tree/main/flower/Dockerfile"
    click Prometheus "https://github.com/aswad546/visiblev8-crawler/blob/main/prometheus/prometheus.yml"
    click Loki "https://github.com/aswad546/visiblev8-crawler/blob/main/loki/loki-config.yaml"
    click Promtail "https://github.com/aswad546/visiblev8-crawler/blob/main/promtail/config.yaml"
    click OTEL "https://github.com/aswad546/visiblev8-crawler/blob/main/otel-collector/otel-collector-config.yaml"

    %% Styles
    classDef service fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef broker fill:#FFCC00,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFE4C4,stroke:#333,stroke-width:1px
    classDef monitor fill:#90EE90,stroke:#333,stroke-width:1px
```
