# misp/misp-galaxy architecture

> Architecture diagram of the GitHub repository misp/misp-galaxy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/misp/misp-galaxy
- Repository: https://github.com/misp/misp-galaxy

## Overview

##### Project type and purpose
- Project type: Open-source knowledge-base generator for MISP (Malware Information Sharing Platform)
- Purpose: Organize and provide “galaxy” clusters (JSON knowledge objects) and vocabularies to attach to MISP events or attributes; generate HTML/PDF documentation; supply CLI tools to transform, validate, and publish data.

##### File structure
Top-level directories and roles:
- clusters, galaxies, vocabularies: static JSON data stores
- schema_*.json: JSON Schema files for validation
- tools/: Python CLI modules and shell scripts to generate clusters, galaxies, docs, and perform validations
– sub-packages: intel-agencies, NER, UN M49, mitre-cti, mkdocs site generator, sigma, tidal-api, etc.
- doc/: image assets for the website
- CONTRIBUTE.md, README.md, .github/workflows, .travis.yml: CI and contribution guidelines
- validate_all.sh, jq_all_the_things.sh: automation scripts to validate data

##### README for additional insights
- Core components: clusters (key JSON), galaxies (higher-level JSON), vocabularies
- Tooling: Python scripts produce JSON, HTML, PDF, and mkdocs static site
- CI pipelines (GitHub Actions, Travis) run tests, linter, validation

##### Main components and interactions
1. clusters/ (JSON cluster files)
2. galaxies/ (JSON galaxy files)
3. vocabularies/ (JSON vocab files)
4. schema_*.json (JSON Schemas)

1. CLI Tools (Python entry points in tools/)
- gen_*.py, csv_to_galaxy.py, add_missing_uuid.py, …
2. Sub-modules
- tools/mitre-cti (data extraction from MITRE CTI feeds)
- tools/mkdocs (site generator)
- tools/tidal-api (fetch data from Tidal Cyber API)
3. Shell scripts
- jq_all_the_things.sh (bulk JSON transformations)
- validate_all.sh (invoke JSON Schema validations)
4. CI pipelines
- GitHub Actions / pytest.yml (unit tests, linting)
- Travis CI (fallback checks)

1. Updated JSON data in clusters/, galaxies/, vocabularies/
2. Generated documentation
- Static site (tools/mkdocs/site → published at misp-galaxy.org)
- README.md index with links
- PDF/HTML overviews

1. MISP Platform (consumes JSON clusters/galaxies via API or manual import)
2. GitHub (source repo, CI triggers)
3. mkdocs (publishes static site)
4. Tidal Cyber API, MITRE CTI feeds (data sources)

##### Relationships & Data Flows
- Processing Layer → Data Store: tools write or update JSON clusters/galaxies
- Processing Layer → Output: mkdocs and README generator produce HTML, PDF
- CI pipelines → Processing Layer: run validation, tests on changes
- External data sources → Processing Layer: tidal-api, mitre-cti scripts fetch data
- Output → Consumers: MISP loads JSON; website hosts docs

##### Architectural patterns & design principles
- File-based pipeline: JSON + schema → Python/CLI transforms → output artifacts
- Modular CLI: each tool focuses on one conversion/generation task
- Separation of concerns: raw data (clusters/galaxies), schema validation, doc generation
- Extensibility: add new data sources by dropping JSON in clusters/ or adding new tool modules

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Store Layer
    subgraph "Data Store Layer"
        DS1((clusters/)):::data
        DS2((galaxies/)):::data
        DS3((vocabularies/)):::data
        DS4["Schemas\n(schema_*.json)"]:::data
    end

    %% Processing & Generation Layer
    subgraph "Processing & Generation Layer"
        subgraph "Core CLI Tools"
            CT1["csv_to_galaxy.py"]:::process
            CT2["add_missing_uuid.py"]:::process
            CT3["gen_mitre.py"]:::process
            CT4["gen_amitt.py"]:::process
            CT5["adoc_galaxy.py"]:::process
            CT6["generate_naics_clusters.py"]:::process
            CT7["update_README_with_index.py"]:::process
        end
        subgraph "Intel Agencies"
            IA1["main.py"]:::process
            IA2["modules/api.py"]:::process
            IA3["modules/intel.py"]:::process
        end
        subgraph "NER"
            NER["extract.py"]:::process
        end
        subgraph "UN M49"
            UN1["generate_region_cluster.py"]:::process
            UN2["region_galaxy.py"]:::process
        end
        subgraph "MITRE CTI"
            MIT1["v1.0/create_attack-pattern_galaxy.py"]:::process
            MIT2["v2.0/create_mitre-enterprise-attack-attack-pattern_galaxy.py"]:::process
        end
        subgraph "Tidal API"
            TA1["main.py"]:::process
            TA2["api/api.py"]:::process
            TA3["models/cluster.py"]:::process
            TA4["models/galaxy.py"]:::process
            TA5["config/*.json"]:::process
        end
        subgraph "MkDocs Site Generator"
            MD1["generator.py"]:::process
            MD2["modules/cluster.py"]:::process
            MD3["modules/galaxy.py"]:::process
            MD4["modules/site.py"]:::process
            MD5["modules/universe.py"]:::process
            MD6["build.sh"]:::process
            MD7["site/"]:::process
        end
        subgraph "Sigma Integration"
            SIG1["sigma-to-galaxy.py"]:::process
            SIG2["update.sh"]:::process
        end
        subgraph "Shell Automation Scripts"
            SH1["jq_all_the_things.sh"]:::process
            SH2["validate_all.sh"]:::process
        end
        subgraph "CI Pipelines"
            CI1[".github/workflows/pytest.yml"]:::process
            CI2[".travis.yml"]:::process
        end
    end

    %% Output/Delivery Layer
    subgraph "Output/Delivery Layer"
        OUT1(("Updated JSON Data")):::output
        OUT2["README.md"]:::output
        OUT3(("Static Site\n(tools/mkdocs/site)")):::output
        OUT4(("PDF/HTML Docs")):::output
    end

    %% External Integrations & Consumers
    subgraph "External Integrations & Consumers"
        EX1((MISP Platform)):::external
        EX2((GitHub)):::external
        EX3((mkdocs)):::external
        EX4((Tidal Cyber API)):::external
        EX5((MITRE CTI Feeds)):::external
    end

    %% Relationships
    DS1 -->|"read"| CT1
    DS2 -->|read| CT1
    DS3 -->|read| CT1
    DS4 -->|validate| SH2
    CT1 -->|write| DS1
    CT2 -->|update| DS1
    CT3 -->|generate| DS1
    IA1 -->|fetch| EX5
    IA1 -->|process| DS1
    NER -->|extract| DS1
    UN1 -->|generate| DS1
    MIT1 -->|fetch| EX5
    MIT1 -->|process| DS1
    TA1 -->|fetch| EX4
    TA1 -->|process| DS1
    SH2 -->|validate| DS4
    MD1 -->|build| OUT3
    CT7 -->|update| OUT2
    CI1 -->|run| CT1
    CI2 -->|run| CT1
    OUT3 -->|host| EX3
    OUT2 -->|source| EX2
    DS1 -->|import| EX1
    OUT4 -->|download| EX1

    %% Click Events
    click DS1 "https://github.com/misp/misp-galaxy/tree/main/clusters/"
    click DS2 "https://github.com/misp/misp-galaxy/tree/main/galaxies/"
    click DS3 "https://github.com/misp/misp-galaxy/tree/main/vocabularies/"
    click DS4 "https://github.com/misp/misp-galaxy/blob/main/schema_clusters.json"
    click DS4 "https://github.com/misp/misp-galaxy/blob/main/schema_galaxies.json"
    click DS4 "https://github.com/misp/misp-galaxy/blob/main/schema_vocabularies.json"
    click DS4 "https://github.com/misp/misp-galaxy/blob/main/schema_misp.json"
    click CT1 "https://github.com/misp/misp-galaxy/blob/main/tools/csv_to_galaxy.py"
    click CT2 "https://github.com/misp/misp-galaxy/blob/main/tools/add_missing_uuid.py"
    click CT3 "https://github.com/misp/misp-galaxy/blob/main/tools/gen_mitre.py"
    click CT4 "https://github.com/misp/misp-galaxy/blob/main/tools/gen_amitt.py"
    click CT5 "https://github.com/misp/misp-galaxy/blob/main/tools/adoc_galaxy.py"
    click CT6 "https://github.com/misp/misp-galaxy/blob/main/tools/generate_naics_clusters.py"
    click CT7 "https://github.com/misp/misp-galaxy/blob/main/tools/update_README_with_index.py"
    click IA1 "https://github.com/misp/misp-galaxy/blob/main/tools/IntelAgencies/main.py"
    click IA2 "https://github.com/misp/misp-galaxy/blob/main/tools/IntelAgencies/modules/api.py"
    click IA3 "https://github.com/misp/misp-galaxy/blob/main/tools/IntelAgencies/modules/intel.py"
    click NER "https://github.com/misp/misp-galaxy/blob/main/tools/NER/extract.py"
    click UN1 "https://github.com/misp/misp-galaxy/blob/main/tools/UN M49/generate_region_cluster.py"
    click UN2 "https://github.com/misp/misp-galaxy/blob/main/tools/UN M49/region_galaxy.py"
    click MIT1 "https://github.com/misp/misp-galaxy/blob/main/tools/mitre-cti/v1.0/create_attack-pattern_galaxy.py"
    click MIT2 "https://github.com/misp/misp-galaxy/blob/main/tools/mitre-cti/v2.0/create_mitre-enterprise-attack-attack-pattern_galaxy.py"
    click TA1 "https://github.com/misp/misp-galaxy/blob/main/tools/tidal-api/main.py"
    click TA2 "https://github.com/misp/misp-galaxy/blob/main/tools/tidal-api/api/api.py"
    click TA3 "https://github.com/misp/misp-galaxy/blob/main/tools/tidal-api/models/cluster.py"
    click TA4 "https://github.com/misp/misp-galaxy/blob/main/tools/tidal-api/models/galaxy.py"
    click TA5 "https://github.com/misp/misp-galaxy/blob/main/tools/tidal-api/config/*.json"
    click MD1 "https://github.com/misp/misp-galaxy/blob/main/tools/mkdocs/generator.py"
    click MD2 "https://github.com/misp/misp-galaxy/blob/main/tools/mkdocs/modules/cluster.py"
    click MD3 "https://github.com/misp/misp-galaxy/blob/main/tools/mkdocs/modules/galaxy.py"
    click MD4 "https://github.com/misp/misp-galaxy/blob/main/tools/mkdocs/modules/site.py"
    click MD5 "https://github.com/misp/misp-galaxy/blob/main/tools/mkdocs/modules/universe.py"
    click MD6 "https://github.com/misp/misp-galaxy/blob/main/tools/mkdocs/build.sh"
    click MD7 "https://github.com/misp/misp-galaxy/tree/main/tools/mkdocs/site/"
    click SIG1 "https://github.com/misp/misp-galaxy/blob/main/tools/sigma/sigma-to-galaxy.py"
    click SIG2 "https://github.com/misp/misp-galaxy/blob/main/tools/sigma/update.sh"
    click SH1 "https://github.com/misp/misp-galaxy/blob/main/jq_all_the_things.sh"
    click SH2 "https://github.com/misp/misp-galaxy/blob/main/validate_all.sh"
    click CI1 "https://github.com/misp/misp-galaxy/blob/main/.github/workflows/pytest.yml"
    click CI2 "https://github.com/misp/misp-galaxy/blob/main/.travis.yml"
    click OUT2 "https://github.com/misp/misp-galaxy/blob/main/README.md"
    click OUT3 "https://github.com/misp/misp-galaxy/tree/main/tools/mkdocs/site/"
    click EX2 "https://github.com/misp/misp-galaxy/tree/main/.github/"
    click EX4 "https://github.com/misp/misp-galaxy/tree/main/tools/tidal-api/"
    click EX5 "https://github.com/misp/misp-galaxy/tree/main/tools/mitre-cti/"

    %% Styles
    classDef data fill:#AED6F1,stroke:#3498DB,color:#154360
    classDef process fill:#ABEBC6,stroke:#27AE60,color:#145A32
    classDef output fill:#F9E79F,stroke:#F1C40F,color:#7D6608
    classDef external fill:#E8DAEF,stroke:#8E44AD,color:#4A235A
```
