# marco-lancini/cartography-queries architecture

> Architecture diagram of the GitHub repository marco-lancini/cartography-queries, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marco-lancini/cartography-queries
- Repository: https://github.com/marco-lancini/cartography-queries

## Overview

##### Project type and purpose
- This repo is a set of “consumers” and “custom queries” for Cartography, an open‑source cloud asset graph tool.
- It provides:
- A JSON file of annotated queries (queries/queries.json)
- An Elasticsearch ingestion consumer (consumers/elasticsearch)
- A Jupyter Notebook consumer (consumers/jupyter_notebooks)

##### File structure
- Top‑level directories:
- consumers/elastic­se­arch – Dockerfiles, Python scripts, K8s manifests, ElastAlert rules, Kibana objects, transforms
- consumers/jupyter_notebooks – Dockerfile, requirements, notebooks, helper connector
- queries – Dockerfile, JSON of queries, query_manager.py
- Configuration and deployment:
- Kustomize and CronJob YAMLs for K8s deployments
- Dockerfiles to containerize Python-based pipelines
- Patterns:
- Two parallel consumer pipelines (ES & Jupyter) reading from the same source (Cartography/Neo4j)
- A layer of custom queries/metadata as input to both consumers

##### Insights from README
- Explains that Cartography writes to Neo4j
- Consumers pull from Neo4j and push to ES or serve data in notebooks
- queries.json holds metadata‑enriched queries used by query_manager.py

##### Main components and relationships
a. Data sources
- AWS, GCP, Azure (external cloud APIs)
- Cartography collector job (outsourced from this repo)
b. Core graph store
- Neo4j database
c. Custom Queries layer
- queries/queries.json + query_manager.py
d. Consumer pipelines
- Elasticsearch consumer:
– elastic_ingestor.py uses elastic_connector.py
– K8s CronJob runs the ingestor on a schedule
– Writes to Elasticsearch cluster
- Jupyter consumer:
– Jupyter Docker image with neo4j_connector.py
– Interactive notebooks for analysis
e. External services
- Elasticsearch cluster
- Kibana UI
- Jupyter web UI

Interaction flow:
1. Cloud APIs → Cartography → Neo4j
2. Neo4j → (a) ES Ingestor → Elasticsearch → Kibana
(b) Jupyter Notebooks → user exploration
3. queries.json feeds into query_manager logic used by both consumers

##### Architectural patterns and principles
- Decoupled pipelines (microservices‑style): each consumer is its own Docker/K8s unit
- Declarative configs (Kustomize, CronJob, Dockerfiles) for reproducible deployment
- Metadata‑driven design: queries.json drives which queries run
- Separation of concerns: ingestion vs. visualization vs. interactive analysis

- Use clear labels for:
- Cartography collector
- Neo4j store
- Query Manager / queries.json
- ES consumer (ingestor, connector, K8s CronJob)
- Elasticsearch cluster & Kibana
- Jupyter consumer (Docker container, notebooks)
- Cloud → Cartography → Neo4j → [ES pipeline → Elasticsearch → Kibana]
- Neo4j → Jupyter container → Notebook UI
- Blue for storage (Neo4j, Elasticsearch)
- Green for compute/services (ingestor, notebooks)
- Orange for config/artifacts (queries.json, ElastAlert rules, K8s manifests)
- Optionally annotate:
- K8s CronJob schedules
- Docker images by folder
- Data formats (JSON queries, transforms, dash exports)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Cloud APIs
    subgraph "Cloud APIs"
        direction TB
        AWS["AWS API"]:::external
        GCP["GCP API"]:::external
        AZ["Azure API"]:::external
    end

    %% Cartography Collector
    Cartography["Cartography Collector Job"]:::external

    %% Core Graph Store
    Neo4j["Neo4j Database"]:::storage

    %% Custom Queries Layer
    subgraph "Custom Queries Layer"
        direction TB
        QueriesJSON["queries/queries.json"]:::config
        QueryManager["query_manager.py"]:::compute
    end

    %% Elasticsearch Consumer Pipeline
    subgraph "Elasticsearch Consumer"
        direction TB
        ESDocker["Dockerfile"]:::config
        ElasticConnector["elastic_connector.py"]:::compute
        ElasticIngestor["elastic_ingestor.py"]:::compute
        CronJob["ingestor-cronjob.yaml"]:::config
        Kustomize_ES["kustomization.yaml"]:::config
        ESIndex["es-index.json"]:::config
        Transforms["transforms/*.json"]:::config
        ElastAlert["elastalert_rules/*.yaml"]:::config
        KibanaDash["kibana dashboards/*.ndjson"]:::config
    end

    %% Jupyter Notebook Consumer Pipeline
    subgraph "Jupyter Notebook Consumer"
        direction TB
        JupyterDocker["Dockerfile"]:::config
        Requirements["requirements.txt"]:::config
        Neo4jConnector["neo4j_connector.py"]:::compute
        NB1["neo4j_connector.ipynb"]:::compute
        NB2["query_builder.ipynb"]:::compute
        NBReport["report_*.ipynb"]:::compute
    end

    %% External Services
    Elasticsearch["Elasticsearch Cluster"]:::storage
    KibanaUI["Kibana UI"]:::external
    JupyterUI["Jupyter Web UI"]:::external

    %% CI/CD Workflows
    subgraph "CI/CD Workflows"
        direction TB
        WF1["docker_queries.yml"]:::config
        WF2["docker_elastic_ingestor.yml"]:::config
        WF3["docker_jupyter.yml"]:::config
        WF4["reusable-docker.yml"]:::config
    end

    %% Data Flows
    AWS --> Cartography
    GCP --> Cartography
    AZ --> Cartography
    Cartography --> Neo4j
    Neo4j --> ElasticIngestor
    Neo4j --> JupyterDocker
    QueriesJSON --> QueryManager
    QueryManager --> ElasticIngestor
    QueryManager --> JupyterDocker

    %% Elasticsearch flow
    ESDocker --> ElasticConnector
    ESDocker --> ElasticIngestor
    CronJob --> ElasticIngestor
    Kustomize_ES --> CronJob
    ElasticIngestor --> Elasticsearch
    ESIndex --> Elasticsearch
    Transforms --> Elasticsearch
    Elasticsearch --> KibanaUI
    ElastAlert --> KibanaUI
    KibanaDash --> KibanaUI

    %% Jupyter flow
    JupyterDocker --> Requirements
    JupyterDocker --> Neo4jConnector
    JupyterDocker --> NB1
    JupyterDocker --> NB2
    JupyterDocker --> NBReport
    NB1 --> JupyterUI
    NB2 --> JupyterUI
    NBReport --> JupyterUI

    %% CI/CD dependencies
    WF1 --> ESDocker
    WF2 --> ESDocker
    WF2 --> JupyterDocker
    WF3 --> JupyterDocker
    WF4 --> ESDocker
    WF4 --> JupyterDocker
    WF4 --> ESDocker

    %% Click Events
    click QueriesJSON "https://github.com/marco-lancini/cartography-queries/blob/main/queries/queries.json"
    click QueryManager "https://github.com/marco-lancini/cartography-queries/blob/main/queries/query_manager.py"

    click ESDocker "https://github.com/marco-lancini/cartography-queries/tree/main/consumers/elasticsearch/docker/Dockerfile"
    click ElasticConnector "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/docker/py/elastic_connector.py"
    click ElasticIngestor "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/docker/py/elastic_ingestor.py"
    click CronJob "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/deployment/ingestor-cronjob.yaml"
    click Kustomize_ES "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/deployment/kustomization.yaml"
    click ESIndex "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/deployment/es-index.json"
    click Transforms "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/transforms/*.json"
    click ElastAlert "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/elastalert_rules/*.yaml"
    click KibanaDash "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/elasticsearch/kibana/dashboards.ndjson"

    click JupyterDocker "https://github.com/marco-lancini/cartography-queries/tree/main/consumers/jupyter_notebooks/Dockerfile"
    click Requirements "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/jupyter_notebooks/requirements.txt"
    click NB1 "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/jupyter_notebooks/notebooks/neo4j_connector.ipynb"
    click NB2 "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/jupyter_notebooks/notebooks/query_builder.ipynb"
    click NBReport "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/jupyter_notebooks/notebooks/report_*.ipynb"
    click Neo4jConnector "https://github.com/marco-lancini/cartography-queries/blob/main/consumers/neo4j_connector.py"

    click WF1 "https://github.com/marco-lancini/cartography-queries/blob/main/.github/workflows/docker_queries.yml"
    click WF2 "https://github.com/marco-lancini/cartography-queries/blob/main/.github/workflows/docker_elastic_ingestor.yml"
    click WF3 "https://github.com/marco-lancini/cartography-queries/blob/main/.github/workflows/docker_jupyter.yml"
    click WF4 "https://github.com/marco-lancini/cartography-queries/blob/main/.github/workflows/reusable-docker.yml"

    %% Styles
    classDef storage fill:#8ecae6,stroke:#023047,stroke-width:1px
    classDef compute fill:#219ebc,stroke:#023047,stroke-width:1px
    classDef config fill:#ffb703,stroke:#fb8500,stroke-width:1px
    classDef external fill:#ccc,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
