# nationalsecurityagency/datawave architecture

> Architecture diagram of the GitHub repository nationalsecurityagency/datawave, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nationalsecurityagency/datawave
- Repository: https://github.com/nationalsecurityagency/datawave

## Overview

##### Main “logical” layers and components
- Apache Accumulo Data Store: Since DataWave leverages Accumulo, include this external dependency as the primary datastore. Both the ingest and query components interact with it. You may also add components for “in-memory Accumulo” (for testing or local modes) as seen in core/in-memory-accumulo.
- Microservices & API Layer: A set of independent service modules (such as those under microservices/services, microservices/starters, and the REST/web services projects under web-services) expose RESTful APIs for operations such as data access (lookup, query, modification), audit, authorization, configuration, metrics reporting, and more. Represent a “Microservices” layer that aggregates these distinct service modules. List key services such as Accumulo service, Audit service, Authorization service, Dictionary service, and Query/Modification service.
- Client Applications and Gateways: Several directories for web-services, client, and examples indicate that there is an API/Web/REST layer which user applications or external clients use to interact with DataWave. This may also include a WebSocket layer. Represent this so that external users hit an API Gateway that then routes to the microservices.

##### Relationships and data flows clearly
- Ingest modules read external raw datasets, perform normalization (data fusion, extracting metadata, tokenization, etc.) and write processed data into Accumulo.
- The Query Engine then reads from Accumulo using specialized iterators and scanners. It further uses query planning (leveraging JEXL/Lucene parser modules and a rules-based optimizer) before returning results.
- The microservices layer coordinates these activities. For instance, the “accumulo” service may be responsible for managing connections and queries, while the “audit” service handles security logging, and the “authorization” and “config” services manage tenant-specific and security configurations.

##### Technologies and frameworks
- Java and Maven (the pom.xml files indicate Maven-based builds).
- Apache Accumulo as the primary datastore.
- RESTful web services (likely built with JAX-RS) and possibly WebSockets.
- Possibly Docker (as seen in the docker and deploy directories), which may be used for containerized deployment in production.
- Iterators and MapReduce jobs – show that the query and ingest pipelines internally use MapReduce and custom Accumulo iterators.

##### Putting it all together

|                    External Clients                 |
|   (Web, REST, WebSockets, JMS, etc.)                |

│
▼
[API Gateway / Web Services Layer]
(web-services, client, examples)
│
┌─────────┴─────────┐
│                   │
▼                   ▼
[Query Engine]       [Microservices Layer]
(core/query,         – Accumulo service
iterators, etc.)      – Audit service
– Authorization, Config,
– Dictionary, Modification,
– Query/Executor, Metrics, etc.
│
▼
[Apache Accumulo Data Store]
▲
│
[Ingestion Pipelines]
– ingest-core, ingest-csv, ingest-json,
ingest-nyctlc, ingest-ssdeep, ingest-wikipedia
│
▼
[External Raw Data Sources]

Additionally, show common/util libraries (core/utils/accumulo-utils,
core/utils/common-utils, core/utils/metadata-utils) as a shared component that is referenced by both the Query Engine and Microservices. Also add a deployment/CI/CD block connected to many modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Layer
    EC("External Clients"):::external
    ExtData("External Raw Data Sources"):::external

    %% API Layer Subgraph
    subgraph "API Layer"
        APIG("API Gateway / Web Services"):::api
        Client("Client Applications & Gateways"):::api
    end

    %% Processing Components
    QueryE("Query Engine"):::service
    Micro("Microservices Layer"):::service
    IngestionP("Ingestion Pipelines"):::ingestion
    Accumulo("Apache Accumulo Data Store"):::db
    Common("Common/Utility Library"):::utility
    Deploy("Deployment & CI/CD"):::deploy

    %% Data Flow Connections
    EC -->|"API_requests"| APIG
    APIG -->|"routes_to"| QueryE
    APIG -->|"routes_to"| Micro
    APIG -->|"serves"| Client
    QueryE -->|"reads_from"| Accumulo
    Micro -->|"reads/writes"| Accumulo
    IngestionP -->|"writes_to"| Accumulo
    ExtData -->|"feeds"| IngestionP
    Common -->|"supports"| QueryE
    Common -->|"supports"| Micro
    Deploy -->|"builds"| APIG
    Deploy -->|"builds"| QueryE
    Deploy -->|"builds"| Micro
    Deploy -->|"builds"| IngestionP

    %% Click Events for Ingestion Pipelines
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/ingest-core"
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/ingest-csv"
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/ingest-json"
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/ingest-nyctlc"
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/ingest-ssdeep"
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/ingest-wikipedia"
    click IngestionP "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/accumulo-extensions"

    %% Click Events for Query Engine
    click QueryE "https://github.com/nationalsecurityagency/datawave/tree/integration/core/query"
    click QueryE "https://github.com/nationalsecurityagency/datawave/tree/integration/warehouse/query-core"

    %% Click Event for Apache Accumulo Data Store (including in-memory mode)
    click Accumulo "https://github.com/nationalsecurityagency/datawave/tree/integration/core/in-memory-accumulo"

    %% Click Events for Microservices & API Layer
    click Micro "https://github.com/nationalsecurityagency/datawave/tree/integration/microservices"
    click Micro "https://github.com/nationalsecurityagency/datawave/tree/integration/web-services"

    %% Click Events for Core Utilities and Common Libraries
    click Common "https://github.com/nationalsecurityagency/datawave/tree/integration/core/utils/accumulo-utils"
    click Common "https://github.com/nationalsecurityagency/datawave/tree/integration/core/utils/common-utils"
    click Common "https://github.com/nationalsecurityagency/datawave/tree/integration/core/utils/metadata-utils"
    click Common "https://github.com/nationalsecurityagency/datawave/tree/integration/core/base-rest-responses"

    %% Click Events for Client Applications and Gateways
    click Client "https://github.com/nationalsecurityagency/datawave/tree/integration/web-services/client"
    click Client "https://github.com/nationalsecurityagency/datawave/tree/integration/web-services/rest-api"
    click Client "https://github.com/nationalsecurityagency/datawave/tree/integration/microservices/starters/datawave"

    %% Styles
    classDef external fill:#FFCCCC,stroke:#CC0000,stroke-width:2px;
    classDef api fill:#CFE2F3,stroke:#1155CC,stroke-width:2px;
    classDef service fill:#D9EAD3,stroke:#38761D,stroke-width:2px;
    classDef db fill:#FFF2CC,stroke:#BF9000,stroke-width:2px;
    classDef ingestion fill:#EAD1DC,stroke:#B35979,stroke-width:2px;
    classDef utility fill:#FFF2CC,stroke:#8E7CC3,stroke-width:2px;
    classDef deploy fill:#D9D2E9,stroke:#8E7CC3,stroke-width:2px;
```
