How vignol-yw/anssi-vulnerability-data-pipeline works
This repository collects CERT-FR security advisories, enriches their referenced CVEs, and stores complete advisory documents in PostgreSQL. The main workflow runs from the CERT-FR RSS feed through advisory extraction and CVE enrichment, with Redis queues between workers; a loader persists the enriched JSONB payload. The tree also contains dbt staging and mart models, although the README describes analytics as future work. Worker source was sampled; dbt SQL was not, so its model dependencies are not drawn.
Main entry points: cve_worker/worker.py CVE enricher · advisorie_worker/worker.py Advisory collector · loader/worker.py PostgreSQL loader
Components 15
Advisory collection
- Advisory collector
src/advisorie_worker/worker.py - Advisory extraction
src/advisorie_worker/worker.py
CVE enrichment
- CVE enricher
src/cve_worker/worker.py
Queues and storage
- Advisory queue
- Enriched CVE queue
- Error queue
- PostgreSQL loader
src/loader/worker.py - Raw advisories
dbt analytics
- Staging models
cve_dbt/models/staging - Dimension models
cve_dbt/models/marts/dimensions - Fact models
cve_dbt/models/marts/facts
Other components
- CERT-FR RSS
- CERT-FR detail API
- CVE Program API
- FIRST EPSS API
Connections 11
- CERT-FR RSS → Advisory collector: provides feed
- Advisory collector → CERT-FR detail API: fetches detail
- Advisory collector → Advisory extraction: dispatches entries
- Advisory extraction → Advisory queue: enqueues advisories
- Advisory extraction → Error queue: records failures
- Advisory queue → CVE enricher: delivers advisories
- CVE enricher → CVE Program API: fetches CVE data
- CVE enricher → FIRST EPSS API: fetches EPSS
- CVE enricher → Enriched CVE queue: enqueues enriched advisories
- Enriched CVE queue → PostgreSQL loader: delivers advisories
- PostgreSQL loader → Raw advisories: upserts JSONB