# nationalsecurityagency/traceanalysis architecture

> Architecture diagram of the GitHub repository nationalsecurityagency/traceanalysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nationalsecurityagency/traceanalysis
- Repository: https://github.com/nationalsecurityagency/traceanalysis

## Overview

##### Project’s Overall Nature
- TraceAnalysis is not a single full-stack application but rather a monorepo housing a suite of independent yet interrelated tools. These include trace generators, trace analyzers, dataflow engines, database management components, and external integrations (e.g., with Ghidra, Unicorn, and other tracer technologies).

##### File Structure
- At the top level you see several directories corresponding to sub-projects such as “container-tracemadness”, “database-manager”, “dynamic-dataflow”, “dynamic-trace”, “ghidra-lifter”, “ghidra-tracemadness”, “strace-spark”, and various “tracer-…” directories (e.g., tracer-pin, tracer-panda, tracer-qemu-user, tracer-unicorn, tracer-icicle).
- Each subproject generally contains its own Cargo configuration (or pyproject.toml for Python-based ones), source code, tests, and even custom Dockerfiles. This suggests a modular design where each tool is developed, built, and deployed independently but falls under the integration umbrella of TraceAnalysis.
- The “docs” directory confirms that extensive documentation is provided and built from source (including a book-style guide that is aggregated for the whole suite).

##### Main Components & Their Roles
a. Trace Sources / Generation:
– Several “tracer-…” projects (e.g., tracer-pin, tracer-panda, tracer-qemu-user, tracer-unicorn, and tracer-icicle) serve as the trace instrumentation and generation tools.
– “container-tracemadness” and related example projects also indicate containerized or script-driven trace generation.

b. Trace Processing and Analysis:
– “dynamic-trace” and “dynamic-dataflow” represent the core processing engines for analysis, manipulation, and transformation of traces.
– They incorporate several libraries (mostly in Rust) to perform operations like analysis, caching, exporting, and even “lifting” (for converting binary traces into higher level representations).

c. Data Management & Storage:
– “database-manager” handles schema definitions and data related to architectures (e.g., opcode and register JSONL files across various architectures) and likely embeds support for ArangoDB (noted in the Ghidra-tracemadness resource files).

d. Integration & External Tools:
– “strace-spark” (with its Python modules) is another tool for parsing and presenting trace data, perhaps in a different format (e.g., CSV, JSONL).

e. Supporting Components:
– “docs” contains documentation assets for the entire suite.
– “examples” houses sample use cases that demonstrate how to run these tools in typical trace analysis workflows.
– Build orchestration is centralized using “.justfile” recipes, Dockerfiles, and CI/CD scripts (e.g., .gitlab-ci.yml).

##### Relationships and Interactions
- The processed trace data flows into the “database-manager” for normalization and storage.
- External or integrated visualization tools (such as those from the “ghidra-…” projects and strace-spark) query and present this data.
- The build system (using justfiles, Docker, and CI scripts) supports independent build and deployment pipelines for each component while ensuring they function as part of the larger suite.

##### Architectural Patterns & Design Principles
- Modularity: Each tool is developed as its own component while adhering to shared design and build conventions.
- Separation of Concerns: Tracing (data collection), processing/analysis, storage, and visualization are clearly separated into different components.
- Multi-language Integration: Many components are written in Rust (dynamic-* and database-manager), while other parts use Java, Python, and even C/C++ (in tracer-pin). Emphasize how technology choices are deployed based on the problem domain.
- Extensibility: The plugin directories under dynamic-dataflow (e.g., for cbranch, fntrack, fpmodels, pointsto, syscalls) illustrate the framework’s extensibility for adding new analysis modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Trace Generation"
        TracerPin["tracer-pin: Instrumentation tool (C/C++)"]:::gen
        TracerPanda["tracer-panda: Instrumentation tool"]:::gen
        TracerQemuUser["tracer-qemu-user: Instrumentation tool"]:::gen
        TracerUnicorn["tracer-unicorn: Instrumentation tool"]:::gen
        TracerIcicle["tracer-icicle: Instrumentation tool"]:::gen
        ContainerTraceMadness["container-tracemadness: Container-based trace generation"]:::gen
        Examples["examples: Sample trace-generation workflows"]:::gen
    end

    subgraph "Trace Processing and Analysis"
        DynamicTrace["dynamic-trace: Core processing engine"]:::proc
        DynamicDataflow["dynamic-dataflow: Dataflow engine with plugin support"]:::proc
    end

    subgraph "Data Management and Storage"
        DatabaseManager["database-manager: Manages trace schemas & storage"]:::storage
    end

    subgraph "Integration and External Tools"
        GhidraLifter["ghidra-lifter: Ghidra lifting integration (Rust)"]:::int
        GhidraTraceMadness["ghidra-tracemadness: Ghidra integration tool (Java/Gradle)"]:::int
        StraceSpark["strace-spark: Trace parsing and presentation (Python)"]:::int
    end

    subgraph "Supporting Infrastructure"
        BuildOrchestration["Build & Deployment: justfiles, Dockerfiles, CI/CD"]:::infra
        Docs["docs: Comprehensive documentation"]:::infra
    end

    %% Data Flow Connections
    TracerPin -->|"generatesTrace"| DynamicTrace
    TracerPanda -->|"generatesTrace"| DynamicTrace
    TracerQemuUser -->|"generatesTrace"| DynamicTrace
    TracerUnicorn -->|"generatesTrace"| DynamicTrace
    TracerIcicle -->|"generatesTrace"| DynamicTrace
    ContainerTraceMadness -->|"generatesTrace"| DynamicTrace
    Examples -->|"demonstratesGeneration"| DynamicTrace

    DynamicTrace -->|"processesTo"| DynamicDataflow
    DynamicDataflow -->|"outputsAnalysis"| DatabaseManager

    %% Integration Connections
    DynamicDataflow -->|"providesData"| GhidraLifter
    DatabaseManager -->|"servesData"| GhidraTraceMadness
    DatabaseManager -->|"servesData"| StraceSpark
    DynamicDataflow -->|"suppliesAnalysis"| StraceSpark

    %% Infrastructure Support Connections
    BuildOrchestration --- TracerPin
    BuildOrchestration --- TracerPanda
    BuildOrchestration --- TracerQemuUser
    BuildOrchestration --- TracerUnicorn
    BuildOrchestration --- TracerIcicle
    BuildOrchestration --- ContainerTraceMadness
    BuildOrchestration --- Examples
    BuildOrchestration --- DynamicTrace
    BuildOrchestration --- DynamicDataflow
    BuildOrchestration --- DatabaseManager
    BuildOrchestration --- GhidraLifter
    BuildOrchestration --- GhidraTraceMadness
    BuildOrchestration --- StraceSpark
    BuildOrchestration --- Docs

    %% Click Events
    click TracerPin "https://github.com/nationalsecurityagency/traceanalysis/tree/main/tracer-pin"
    click TracerPanda "https://github.com/nationalsecurityagency/traceanalysis/tree/main/tracer-panda"
    click TracerQemuUser "https://github.com/nationalsecurityagency/traceanalysis/tree/main/tracer-qemu-user"
    click TracerUnicorn "https://github.com/nationalsecurityagency/traceanalysis/tree/main/tracer-unicorn"
    click TracerIcicle "https://github.com/nationalsecurityagency/traceanalysis/tree/main/tracer-icicle"
    click ContainerTraceMadness "https://github.com/nationalsecurityagency/traceanalysis/tree/main/container-tracemadness"
    click Examples "https://github.com/nationalsecurityagency/traceanalysis/tree/main/examples"
    click DynamicTrace "https://github.com/nationalsecurityagency/traceanalysis/tree/main/dynamic-trace"
    click DynamicDataflow "https://github.com/nationalsecurityagency/traceanalysis/tree/main/dynamic-dataflow"
    click DatabaseManager "https://github.com/nationalsecurityagency/traceanalysis/tree/main/database-manager"
    click GhidraLifter "https://github.com/nationalsecurityagency/traceanalysis/tree/main/ghidra-lifter"
    click GhidraTraceMadness "https://github.com/nationalsecurityagency/traceanalysis/tree/main/ghidra-tracemadness"
    click StraceSpark "https://github.com/nationalsecurityagency/traceanalysis/tree/main/strace-spark"
    click BuildOrchestration "https://github.com/nationalsecurityagency/traceanalysis/blob/main/.justfile"
    click Docs "https://github.com/nationalsecurityagency/traceanalysis/tree/main/docs"

    %% Styles
    classDef gen fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef proc fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef storage fill:#f9d976,stroke:#333,stroke-width:2px;
    classDef int fill:#ffcccb,stroke:#333,stroke-width:2px;
    classDef infra fill:#d3d3d3,stroke:#333,stroke-width:2px;
```
