How nationalsecurityagency/datawave works
Main “logical” layers and components
• Apache Accumulo Data Store: Since DataWave leverages Accumulo, include this external dependency as the primary datastore. Both the ingest and query components interact with it. You may also add components for “in-memory Accumulo” (for testing or local modes) as seen in core/in-memory-accumulo.
• Microservices & API Layer: A set of independent service modules (such as those under microservices/services, microservices/starters, and the REST/web services projects under web-services) expose RESTful APIs for operations such as data access (lookup, query, modification), audit, authorization, configuration, metrics reporting, and more. Represent a “Microservices” layer that aggregates these distinct service modules. List key services such as Accumulo service, Audit service, Authorization service, Dictionary service, and Query/Modification service.
• Client Applications and Gateways: Several directories for web-services, client, and examples indicate that there is an API/Web/REST layer which user applications or external clients use to interact with DataWave. This may also include a WebSocket layer. Represent this so that external users hit an API Gateway that then routes to the microservices.
Relationships and data flows clearly
• Ingest modules read external raw datasets, perform normalization (data fusion, extracting metadata, tokenization, etc.) and write processed data into Accumulo.
• The Query Engine then reads from Accumulo using specialized iterators and scanners. It further uses query planning (leveraging JEXL/Lucene parser modules and a rules-based optimizer) before returning results.
• The microservices layer coordinates these activities. For instance, the “accumulo” service may be responsible for managing connections and queries, while the “audit” service handles security logging, and the “authorization” and “config” services manage tenant-specific and security configurations.
Technologies and frameworks
• Java and Maven (the pom.xml files indicate Maven-based builds).
• Apache Accumulo as the primary datastore.
• RESTful web services (likely built with JAX-RS) and possibly WebSockets.
• Possibly Docker (as seen in the docker and deploy directories), which may be used for containerized deployment in production.
• Iterators and MapReduce jobs – show that the query and ingest pipelines internally use MapReduce and custom Accumulo iterators.
Putting it all together
| External Clients |
| (Web, REST, WebSockets, JMS, etc.) |
│
▼
[API Gateway / Web Services Layer]
(web-services, client, examples)
│
┌─────────┴─────────┐
│ │
▼ ▼
[Query Engine] [Microservices Layer]
(core/query, – Accumulo service
iterators, etc.) – Audit service
– Authorization, Config,
– Dictionary, Modification,
– Query/Executor, Metrics, etc.
│
▼
[Apache Accumulo Data Store]
▲
│
[Ingestion Pipelines]
– ingest-core, ingest-csv, ingest-json,
ingest-nyctlc, ingest-ssdeep, ingest-wikipedia
│
▼
[External Raw Data Sources]
Additionally, show common/util libraries (core/utils/accumulo-utils,
core/utils/common-utils, core/utils/metadata-utils) as a shared component that is referenced by both the Query Engine and Microservices. Also add a deployment/CI/CD block connected to many modules.