How marco-lancini/cartography-queries works
Project type and purpose
• This repo is a set of “consumers” and “custom queries” for Cartography, an open‑source cloud asset graph tool.
• It provides:
• A JSON file of annotated queries (queries/queries.json)
• An Elasticsearch ingestion consumer (consumers/elasticsearch)
• A Jupyter Notebook consumer (consumers/jupyter_notebooks)
File structure
• Top‑level directories:
• consumers/elasticsearch – Dockerfiles, Python scripts, K8s manifests, ElastAlert rules, Kibana objects, transforms
• consumers/jupyter_notebooks – Dockerfile, requirements, notebooks, helper connector
• queries – Dockerfile, JSON of queries, query_manager.py
• Configuration and deployment:
• Kustomize and CronJob YAMLs for K8s deployments
• Dockerfiles to containerize Python-based pipelines
• Patterns:
• Two parallel consumer pipelines (ES & Jupyter) reading from the same source (Cartography/Neo4j)
• A layer of custom queries/metadata as input to both consumers
Insights from README
• Explains that Cartography writes to Neo4j
• Consumers pull from Neo4j and push to ES or serve data in notebooks
• queries.json holds metadata‑enriched queries used by query_manager.py
Main components and relationships
a. Data sources
• AWS, GCP, Azure (external cloud APIs)
• Cartography collector job (outsourced from this repo)
b. Core graph store
• Neo4j database
c. Custom Queries layer
• queries/queries.json + query_manager.py
d. Consumer pipelines
• Elasticsearch consumer:
– elastic_ingestor.py uses elastic_connector.py
– K8s CronJob runs the ingestor on a schedule
– Writes to Elasticsearch cluster
• Jupyter consumer:
– Jupyter Docker image with neo4j_connector.py
– Interactive notebooks for analysis
e. External services
• Elasticsearch cluster
• Kibana UI
• Jupyter web UI
Interaction flow:
1. Cloud APIs → Cartography → Neo4j
2. Neo4j → (a) ES Ingestor → Elasticsearch → Kibana
(b) Jupyter Notebooks → user exploration
3. queries.json feeds into query_manager logic used by both consumers
Architectural patterns and principles
• Decoupled pipelines (microservices‑style): each consumer is its own Docker/K8s unit
• Declarative configs (Kustomize, CronJob, Dockerfiles) for reproducible deployment
• Metadata‑driven design: queries.json drives which queries run
• Separation of concerns: ingestion vs. visualization vs. interactive analysis
• Use clear labels for:
• Cartography collector
• Neo4j store
• Query Manager / queries.json
• ES consumer (ingestor, connector, K8s CronJob)
• Elasticsearch cluster & Kibana
• Jupyter consumer (Docker container, notebooks)
• Cloud → Cartography → Neo4j → [ES pipeline → Elasticsearch → Kibana]
• Neo4j → Jupyter container → Notebook UI
• Blue for storage (Neo4j, Elasticsearch)
• Green for compute/services (ingestor, notebooks)
• Orange for config/artifacts (queries.json, ElastAlert rules, K8s manifests)
• Optionally annotate:
• K8s CronJob schedules
• Docker images by folder
• Data formats (JSON queries, transforms, dash exports)