# yagikota/danewebperf architecture

> Architecture diagram of the GitHub repository yagikota/danewebperf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yagikota/danewebperf
- Repository: https://github.com/yagikota/danewebperf

## Overview

##### Project Type and Purpose
- DANEWebPerf is a containerized benchmarking tool (full-stack CLI + microservices style) written primarily in Go, augmented by Python dataset scripts.
- It orchestrates three Dockerized components—Firefox (with Selenium), Unbound (DNS cache), and Let’s DANE (DANE proxy)—to measure how DANE impacts web page load times under four scenarios.
- Results are run on EC2 and stored in an S3 bucket.

a. Orchestrator (EC2 host)
– Go CLI commands under `cmd/` (dane-check, pageloadtime, etc.)
– Shell scripts (`install.sh`, `prepare.sh`, `start.sh`)
– Dataset preparers in `dataset/`
– Docker daemon and AWS CLI client
b. Docker Network (internal virtual network)
– Firefox Container
- Headless Firefox + Selenium (requirements.txt, pageload_measure.py)
– Unbound Container
- DNS cache server (Dockerfile, unbound.conf)
– Let’s DANE Container
- DANE-EE(3) proxy (Go code under `docker/firefox/letsdane/`)
c. External Services
– Internet (target websites; DNS root/authoritative servers)
– AWS S3 (results storage bucket)
d. Build Artifacts
– Docker images (built from `docker/firefox/Dockerfile.firefox` and `docker/unbound/Dockerfile.*`)
– Go binaries (built from `cmd/*` via Go modules `go.mod`/`go.sum`)

##### Relationships and Data Flows
- Orchestrator → Docker API: “docker build” and “docker run” commands (prepare.sh, start.sh)
- Docker Network internal traffic:
- Firefox → (direct) Unbound → public Internet → Unbound → Firefox
– Scenarios w/o DANE
- Firefox → Let’s DANE → Unbound → public Internet → Unbound → Let’s DANE → Firefox
– Scenarios w/ DANE
- Orchestrator → AWS S3: uploads HAR, PCAP, CSV results
- Dataset scripts → Orchestrator: domain lists and scraped HTML/CSV data

##### Architectural Patterns & Design Principles
- Microservices style: each major function isolated in its own container
- Clear separation of concerns:
- Measurement controller (Go CLI + shell scripts)
- Data collection agents (Firefox, Unbound, Let’s DANE)
- Dataset preparation (Python)
- Reproducible builds: Dockerfiles under `docker/`, Go modules, Python requirements

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Host Orchestrator
    subgraph "Host: Orchestrator (EC2)" 
        direction TB
        Orchestrator_CLI["Go CLI (cmd/)"]:::orchestrator
        InstallSh["install.sh"]:::orchestrator
        PrepareSh["prepare.sh"]:::orchestrator
        StartSh["start.sh"]:::orchestrator
        GoModules["Go Modules (go.mod/go.sum)"]:::orchestrator
        AWSCLI["AWS CLI Client"]:::orchestrator
        DockerDaemon["Docker Daemon"]:::orchestrator
    end

    %% Dataset Preparation
    subgraph "Dataset Preparation" 
        direction TB
        DatasetDir["dataset/"]:::dataset
        PrepareDataset["prepare-dataset.sh"]:::dataset
        ScrapePy["scrape.py"]:::dataset
        SaveHtml["save-html.py"]:::dataset
        UsageCsv["usage3-domain-csv.py"]:::dataset
    end

    %% Docker Network
    subgraph "Docker Network" 
        direction TB
        subgraph "Firefox Container" 
            direction TB
            FirefoxContainer["Firefox Container\n(Firefox 67.0.1 + Selenium 3.141.0)"]:::container
            FF_Dockerfile["Dockerfile.firefox"]:::container
            FF_Entrypoint["entrypoint.sh"]:::container
            FF_Script["pageload_measure.py"]:::container
            FF_Reqs["requirements.txt"]:::container
            FF_Policies["policies.json"]:::container
        end
        subgraph "Unbound Container" 
            direction TB
            UnboundCache["Unbound Cache"]:::container
            U_CacheDF["Dockerfile.unbound-with-cache"]:::container
            U_NoCacheDF["Dockerfile.unbound-without-cache"]:::container
            U_Conf["unbound.conf"]:::container
            U_StartCache["start-with-cache.sh"]:::container
            U_StartNoCache["start-without-cache.sh"]:::container
        end
        subgraph "Let's DANE Container" 
            direction TB
            LetsDANE["Let's DANE Proxy"]:::container
            LD_Dir["docker/firefox/letsdane/"]:::container
            LD_DF["Dockerfile.letsdane"]:::container
            LD_Entrypoint["entrypoint.sh"]:::container
            LD_Proxy["proxy/"]:::container
            LD_Resolver["resolver/"]:::container
            LD_Certs["cert.crt, cert.key, ca/"]:::container
        end
    end

    %% External Services
    subgraph "External Services"
        direction TB
        Internet["Public Internet\n(DNS root/auth servers)"]:::external
        S3["AWS S3 Bucket"]:::s3
    end

    %% Flows
    DatasetDir -->|provides domains| Orchestrator_CLI
    Orchestrator_CLI -->|build/run| DockerDaemon
    Orchestrator_CLI -->|upload results| S3
    DockerDaemon --> FirefoxContainer
    DockerDaemon --> UnboundCache
    DockerDaemon --> LetsDANE

    %% Measurement Flows without DANE
    FirefoxContainer -->|"DNS lookup"| UnboundCache
    UnboundCache -->|"to Internet"| Internet
    Internet -->|"to cache"| UnboundCache
    UnboundCache -->|"to browser"| FirefoxContainer

    %% Measurement Flows with DANE
    FirefoxContainer -->|"DANE proxy"| LetsDANE
    LetsDANE -->|"DNS lookup"| UnboundCache
    UnboundCache -->|"to Internet"| Internet
    Internet -->|"to cache"| UnboundCache
    UnboundCache -->|"via proxy"| LetsDANE
    LetsDANE -->|"to browser"| FirefoxContainer

    Orchestrator_CLI -->|controls| FF_Script

    %% Click Events - Orchestrator
    click Orchestrator_CLI "https://github.com/yagikota/danewebperf/tree/main/cmd/"
    click InstallSh "https://github.com/yagikota/danewebperf/blob/main/install.sh"
    click PrepareSh "https://github.com/yagikota/danewebperf/blob/main/prepare.sh"
    click StartSh "https://github.com/yagikota/danewebperf/blob/main/start.sh"
    click GoModules "https://github.com/yagikota/danewebperf/blob/main/go.mod"
    click AWSCLI "https://github.com/yagikota/danewebperf/tree/main/cmd/"
    click DockerDaemon "https://github.com/yagikota/danewebperf/tree/main/cmd/"

    %% Click Events - Dataset
    click DatasetDir "https://github.com/yagikota/danewebperf/tree/main/dataset/"
    click PrepareDataset "https://github.com/yagikota/danewebperf/blob/main/dataset/prepare-dataset.sh"
    click ScrapePy "https://github.com/yagikota/danewebperf/blob/main/dataset/scrape.py"
    click SaveHtml "https://github.com/yagikota/danewebperf/blob/main/dataset/save-html.py"
    click UsageCsv "https://github.com/yagikota/danewebperf/blob/main/dataset/usage3-domain-csv.py"

    %% Click Events - Firefox Container
    click FF_Dockerfile "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/Dockerfile.firefox"
    click FF_Entrypoint "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/entrypoint.sh"
    click FF_Script "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/pageload_measure.py"
    click FF_Reqs "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/requirements.txt"
    click FF_Policies "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/policies.json"

    %% Click Events - Unbound Container
    click U_CacheDF "https://github.com/yagikota/danewebperf/blob/main/docker/unbound/Dockerfile.unbound-with-cache"
    click U_NoCacheDF "https://github.com/yagikota/danewebperf/blob/main/docker/unbound/Dockerfile.unbound-without-cache"
    click U_Conf "https://github.com/yagikota/danewebperf/blob/main/docker/unbound/unbound.conf"
    click U_StartCache "https://github.com/yagikota/danewebperf/blob/main/docker/unbound/start-with-cache.sh"
    click U_StartNoCache "https://github.com/yagikota/danewebperf/blob/main/docker/unbound/start-without-cache.sh"

    %% Click Events - Let's DANE Container
    click LD_Dir "https://github.com/yagikota/danewebperf/tree/main/docker/firefox/letsdane/"
    click LD_DF "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/letsdane/Dockerfile.letsdane"
    click LD_Entrypoint "https://github.com/yagikota/danewebperf/blob/main/docker/firefox/letsdane/entrypoint.sh"
    click LD_Proxy "https://github.com/yagikota/danewebperf/tree/main/docker/firefox/letsdane/proxy/"
    click LD_Resolver "https://github.com/yagikota/danewebperf/tree/main/docker/firefox/letsdane/resolver/"
    click LD_Certs "https://github.com/yagikota/danewebperf/tree/main/docker/firefox/letsdane/ca/"

    classDef orchestrator fill:#FFA500,stroke:#333,stroke-width:1px
    classDef container fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef dataset fill:#98FB98,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef s3 fill:#90EE90,stroke:#333,stroke-width:1px
```
