# vignol-yw/anssi-vulnerability-data-pipeline architecture

> Architecture diagram of the GitHub repository vignol-yw/anssi-vulnerability-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vignol-yw/anssi-vulnerability-data-pipeline
- Repository: https://github.com/vignol-yw/anssi-vulnerability-data-pipeline
- Video tour (about a minute): https://gitdiagram.com/vignol-yw/anssi-vulnerability-data-pipeline/video

## Overview

This repository collects CERT-FR security advisories, enriches their referenced CVEs, and stores complete advisory documents in PostgreSQL. The main workflow runs from the CERT-FR RSS feed through advisory extraction and CVE enrichment, with Redis queues between workers; a loader persists the enriched JSONB payload. The tree also contains dbt staging and mart models, although the README describes analytics as future work. Worker source was sampled; dbt SQL was not, so its model dependencies are not drawn.

Main entry points: [`src/cve_worker/worker.py`](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/cve_worker/worker.py) (CVE enricher), [`src/advisorie_worker/worker.py`](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/advisorie_worker/worker.py) (Advisory collector), [`src/loader/worker.py`](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/loader/worker.py) (PostgreSQL loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Advisory collection"]
  node_collector["Advisory collector<br/>[worker.py]"]
  node_advisory_extract["Advisory extraction<br/>[worker.py]"]
end

subgraph group_enrichment["CVE enrichment"]
  node_cve_enricher["CVE enricher<br/>[worker.py]"]
end

subgraph group_transport["Queues and storage"]
  node_advisories_queue["Advisory queue"]
  node_cve_queue["Enriched CVE queue"]
  node_error_queue["Error queue"]
  node_loader["PostgreSQL loader<br/>[worker.py]"]
  node_raw_advisories[("Raw advisories")]
end

subgraph group_analytics["dbt analytics"]
  node_staging_models["Staging models"]
  node_dimension_models["Dimension models"]
  node_fact_models["Fact models"]
end

node_certfr_feed(("CERT-FR RSS"))
node_certfr_detail["CERT-FR detail API"]
node_cve_program["CVE Program API"]
node_first_epss["FIRST EPSS API"]

node_certfr_feed -->|"provides feed"| node_collector
node_collector -->|"fetches detail"| node_certfr_detail
node_collector -->|"dispatches entries"| node_advisory_extract
node_advisory_extract -->|"enqueues advisories"| node_advisories_queue
node_advisory_extract -->|"records failures"| node_error_queue
node_advisories_queue -->|"delivers advisories"| node_cve_enricher
node_cve_enricher -->|"fetches CVE data"| node_cve_program
node_cve_enricher -->|"fetches EPSS"| node_first_epss
node_cve_enricher -->|"enqueues enriched advisories"| node_cve_queue
node_cve_queue -->|"delivers advisories"| node_loader
node_loader -->|"upserts JSONB"| node_raw_advisories

click node_collector "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/advisorie_worker/worker.py"
click node_advisory_extract "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/advisorie_worker/worker.py"
click node_cve_enricher "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/cve_worker/worker.py"
click node_loader "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/loader/worker.py"
click node_staging_models "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/tree/main/cve_dbt/models/staging"
click node_dimension_models "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/tree/main/cve_dbt/models/marts/dimensions"
click node_fact_models "https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/tree/main/cve_dbt/models/marts/facts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collector,node_advisory_extract toneBlue
class node_cve_enricher toneAmber
class node_advisories_queue,node_cve_queue,node_error_queue,node_loader,node_raw_advisories,node_certfr_detail,node_cve_program,node_first_epss toneMint
class node_staging_models,node_dimension_models,node_fact_models toneRose
class node_certfr_feed toneIndigo
```

## Components

### Advisory collection

- [Advisory collector](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/advisorie_worker/worker.py): `src/advisorie_worker/worker.py`
- [Advisory extraction](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/advisorie_worker/worker.py): `src/advisorie_worker/worker.py`

### CVE enrichment

- [CVE enricher](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/cve_worker/worker.py): `src/cve_worker/worker.py`

### Queues and storage

- **Advisory queue**
- **Enriched CVE queue**
- **Error queue**
- [PostgreSQL loader](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/blob/main/src/loader/worker.py): `src/loader/worker.py`
- **Raw advisories**

### dbt analytics

- [Staging models](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/tree/main/cve_dbt/models/staging): `cve_dbt/models/staging`
- [Dimension models](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/tree/main/cve_dbt/models/marts/dimensions): `cve_dbt/models/marts/dimensions`
- [Fact models](https://github.com/vignol-yw/anssi-vulnerability-data-pipeline/tree/main/cve_dbt/models/marts/facts): `cve_dbt/models/marts/facts`

### Other components

- **CERT-FR RSS**
- **CERT-FR detail API**
- **CVE Program API**
- **FIRST EPSS API**

## Connections

- CERT-FR RSS → Advisory collector: provides feed
- Advisory collector → CERT-FR detail API: fetches detail
- Advisory collector → Advisory extraction: dispatches entries
- Advisory extraction → Advisory queue: enqueues advisories
- Advisory extraction → Error queue: records failures
- Advisory queue → CVE enricher: delivers advisories
- CVE enricher → CVE Program API: fetches CVE data
- CVE enricher → FIRST EPSS API: fetches EPSS
- CVE enricher → Enriched CVE queue: enqueues enriched advisories
- Enriched CVE queue → PostgreSQL loader: delivers advisories
- PostgreSQL loader → Raw advisories: upserts JSONB