How misp/misp-galaxy works
Project type and purpose
• Project type: Open-source knowledge-base generator for MISP (Malware Information Sharing Platform)
• Purpose: Organize and provide “galaxy” clusters (JSON knowledge objects) and vocabularies to attach to MISP events or attributes; generate HTML/PDF documentation; supply CLI tools to transform, validate, and publish data.
File structure
Top-level directories and roles:
• clusters, galaxies, vocabularies: static JSON data stores
• schema_*.json: JSON Schema files for validation
• tools/: Python CLI modules and shell scripts to generate clusters, galaxies, docs, and perform validations
– sub-packages: intel-agencies, NER, UN M49, mitre-cti, mkdocs site generator, sigma, tidal-api, etc.
• doc/: image assets for the website
• CONTRIBUTE.md, README.md, .github/workflows, .travis.yml: CI and contribution guidelines
• validate_all.sh, jq_all_the_things.sh: automation scripts to validate data
README for additional insights
• Core components: clusters (key JSON), galaxies (higher-level JSON), vocabularies
• Tooling: Python scripts produce JSON, HTML, PDF, and mkdocs static site
• CI pipelines (GitHub Actions, Travis) run tests, linter, validation
Main components and interactions
1. clusters/ (JSON cluster files)
2. galaxies/ (JSON galaxy files)
3. vocabularies/ (JSON vocab files)
4. schema_*.json (JSON Schemas)
1. CLI Tools (Python entry points in tools/)
• gen_*.py, csv_to_galaxy.py, add_missing_uuid.py, …
2. Sub-modules
• tools/mitre-cti (data extraction from MITRE CTI feeds)
• tools/mkdocs (site generator)
• tools/tidal-api (fetch data from Tidal Cyber API)
3. Shell scripts
• jq_all_the_things.sh (bulk JSON transformations)
• validate_all.sh (invoke JSON Schema validations)
4. CI pipelines
• GitHub Actions / pytest.yml (unit tests, linting)
• Travis CI (fallback checks)
1. Updated JSON data in clusters/, galaxies/, vocabularies/
2. Generated documentation
• Static site (tools/mkdocs/site → published at misp-galaxy.org)
• README.md index with links
• PDF/HTML overviews
1. MISP Platform (consumes JSON clusters/galaxies via API or manual import)
2. GitHub (source repo, CI triggers)
3. mkdocs (publishes static site)
4. Tidal Cyber API, MITRE CTI feeds (data sources)
Relationships & Data Flows
• Processing Layer → Data Store: tools write or update JSON clusters/galaxies
• Processing Layer → Output: mkdocs and README generator produce HTML, PDF
• CI pipelines → Processing Layer: run validation, tests on changes
• External data sources → Processing Layer: tidal-api, mitre-cti scripts fetch data
• Output → Consumers: MISP loads JSON; website hosts docs
Architectural patterns & design principles
• File-based pipeline: JSON + schema → Python/CLI transforms → output artifacts
• Modular CLI: each tool focuses on one conversion/generation task
• Separation of concerns: raw data (clusters/galaxies), schema validation, doc generation
• Extensibility: add new data sources by dropping JSON in clusters/ or adding new tool modules