# googlecloudplatform/bq-otel-exporter architecture

> Architecture diagram of the GitHub repository googlecloudplatform/bq-otel-exporter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/bq-otel-exporter
- Repository: https://github.com/googlecloudplatform/bq-otel-exporter

## Overview

This repository is an OpenTelemetry Collector exporter that transports traces, logs, and metrics into BigQuery, with optional payload offload and quarantine. Its main path is Collector input → exporter processing and encoding → BigQuery append. Separate commands support archive backfill, quarantine replay, destination validation, and load benchmarking. The README documents downstream views and alerting, but those systems are outside this source tree. Relationships for unsampled internals are included only when supported by the source index or README.

Main entry points: [`exporter.go`](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter.go) (Signal exporter), [`internal/cli/backfill/backfill.go`](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/backfill/backfill.go) (Archive backfill), [`internal/offload/offload.go`](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/offload/offload.go) (Payload offload), [`internal/cli/quarantinereplay/replay.go`](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/quarantinereplay/replay.go) (Quarantine replay)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_export["Export runtime"]
  node_exporter["Signal exporter<br/>[exporter.go]"]
  node_signal_maps["Signal mappings<br/>[exporter_traces.go]"]
  node_log_maps["Log mappings<br/>[exporter_logs.go]"]
  node_metric_maps["Metric mappings"]
end

subgraph group_processing["Record processing"]
  node_row_transform["Row transforms<br/>[rows.go]"]
  node_metric_transform["Metric transforms<br/>[metrics.go]"]
  node_wire_encoder["Wire encoding<br/>[wire_encoder.go]"]
  node_offload["Payload offload<br/>[offload.go]"]
  node_redaction["Redaction<br/>[redact.go]"]
end

subgraph group_operations["Operational commands"]
  node_backfill_cmd["Backfill command<br/>[main.go]"]
  node_backfill["Archive backfill<br/>[backfill.go]"]
  node_replay_cmd["Replay command<br/>[main.go]"]
  node_replay["Quarantine replay<br/>[replay.go]"]
  node_validate_cmd["Validation command<br/>[main.go]"]
  node_validator["Dataset validation<br/>[validate.go]"]
  node_bench_cmd["Benchmark command<br/>[main.go]"]
  node_bench_runner["Benchmark runner<br/>[runner.go]"]
end

subgraph group_destination["BigQuery destination"]
  node_bq_writer["Storage writer<br/>[writer.go]"]
  node_bigquery[("BigQuery tables")]
end

node_collector(("Collector"))
node_cloud_storage[("Cloud Storage")]
node_analytics["Views and alerts"]

node_collector -->|"sends OTLP"| node_exporter
node_exporter -->|"maps traces"| node_signal_maps
node_exporter -->|"maps logs"| node_log_maps
node_exporter -->|"maps metrics"| node_metric_maps
node_exporter -->|"transforms records"| node_row_transform
node_exporter -->|"transforms metrics"| node_metric_transform
node_exporter -->|"encodes rows"| node_wire_encoder
node_exporter -->|"configures redaction"| node_redaction
node_exporter -->|"configures offload"| node_offload
node_exporter -->|"writes batches"| node_bq_writer
node_bq_writer -->|"appends rows"| node_bigquery
node_offload -.->|"stores payloads"| node_cloud_storage
node_bigquery -->|"feeds views"| node_analytics
node_backfill_cmd -->|"runs backfill"| node_backfill
node_backfill -.->|"appends history"| node_bigquery
node_cloud_storage -->|"provides archives"| node_backfill
node_replay_cmd -->|"runs replay"| node_replay
node_replay -.->|"reads quarantine"| node_bigquery
node_validate_cmd -->|"runs checks"| node_validator
node_validator -.->|"checks destination"| node_bigquery
node_bench_cmd -->|"runs benchmark"| node_bench_runner
node_bench_runner -->|"creates exporters"| node_exporter

click node_exporter "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter.go"
click node_signal_maps "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter_traces.go"
click node_log_maps "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter_logs.go"
click node_metric_maps "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter_metrics.go"
click node_row_transform "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/transform/rows.go"
click node_metric_transform "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/transform/metrics.go"
click node_wire_encoder "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/protoenc/wire_encoder.go"
click node_offload "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/offload/offload.go"
click node_redaction "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/redact/redact.go"
click node_bq_writer "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/bigquery/writer.go"
click node_backfill_cmd "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-otel-backfill/main.go"
click node_backfill "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/backfill/backfill.go"
click node_replay_cmd "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-quarantine-replay/main.go"
click node_replay "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/quarantinereplay/replay.go"
click node_validate_cmd "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-otel-validate/main.go"
click node_validator "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/validate/validate.go"
click node_bench_cmd "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-otel-testbench/main.go"
click node_bench_runner "https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/testbench/runner.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_exporter,node_signal_maps,node_log_maps,node_metric_maps,node_analytics toneBlue
class node_row_transform,node_metric_transform,node_wire_encoder,node_offload,node_redaction,node_cloud_storage toneAmber
class node_backfill_cmd,node_backfill,node_replay_cmd,node_replay,node_validate_cmd,node_validator,node_bench_cmd,node_bench_runner toneMint
class node_bq_writer,node_bigquery toneRose
class node_collector toneIndigo
```

## Components

### Export runtime

- [Signal exporter](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter.go): `exporter.go`
- [Signal mappings](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter_traces.go): `exporter_traces.go`
- [Log mappings](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter_logs.go): `exporter_logs.go`
- [Metric mappings](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/exporter_metrics.go): `exporter_metrics.go`

### Record processing

- [Row transforms](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/transform/rows.go): `internal/transform/rows.go`
- [Metric transforms](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/transform/metrics.go): `internal/transform/metrics.go`
- [Wire encoding](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/protoenc/wire_encoder.go): `internal/protoenc/wire_encoder.go`
- [Payload offload](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/offload/offload.go): `internal/offload/offload.go`
- [Redaction](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/redact/redact.go): `internal/redact/redact.go`

### Operational commands

- [Backfill command](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-otel-backfill/main.go): `cmd/bq-otel-backfill/main.go`
- [Archive backfill](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/backfill/backfill.go): `internal/cli/backfill/backfill.go`
- [Replay command](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-quarantine-replay/main.go): `cmd/bq-quarantine-replay/main.go`
- [Quarantine replay](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/quarantinereplay/replay.go): `internal/cli/quarantinereplay/replay.go`
- [Validation command](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-otel-validate/main.go): `cmd/bq-otel-validate/main.go`
- [Dataset validation](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/validate/validate.go): `internal/cli/validate/validate.go`
- [Benchmark command](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/cmd/bq-otel-testbench/main.go): `cmd/bq-otel-testbench/main.go`
- [Benchmark runner](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/cli/testbench/runner.go): `internal/cli/testbench/runner.go`

### BigQuery destination

- [Storage writer](https://github.com/googlecloudplatform/bq-otel-exporter/blob/main/internal/bigquery/writer.go): `internal/bigquery/writer.go`
- **BigQuery tables**

### Other components

- **Collector**
- **Cloud Storage**
- **Views and alerts**

## Connections

- Collector → Signal exporter: sends OTLP (evidence: `README.md`)
- Signal exporter → Signal mappings: maps traces (evidence: `exporter.go`)
- Signal exporter → Log mappings: maps logs (evidence: `exporter.go`)
- Signal exporter → Metric mappings: maps metrics (evidence: `exporter.go`)
- Signal exporter → Row transforms: transforms records (evidence: `exporter.go`)
- Signal exporter → Metric transforms: transforms metrics (evidence: `exporter.go`)
- Signal exporter → Wire encoding: encodes rows (evidence: `exporter.go`)
- Signal exporter → Redaction: configures redaction (evidence: `exporter.go`)
- Signal exporter → Payload offload: configures offload (evidence: `exporter.go`)
- Signal exporter → Storage writer: writes batches (evidence: `exporter.go`)
- Storage writer → BigQuery tables: appends rows (evidence: `README.md`)
- Payload offload → Cloud Storage: stores payloads (evidence: `internal/offload/offload.go`)
- BigQuery tables → Views and alerts: feeds views (evidence: `README.md`)
- Backfill command → Archive backfill: runs backfill (evidence: `cmd/bq-otel-backfill/main.go`)
- Archive backfill → BigQuery tables: appends history
- Cloud Storage → Archive backfill: provides archives
- Replay command → Quarantine replay: runs replay (evidence: `cmd/bq-quarantine-replay/main.go`)
- Quarantine replay → BigQuery tables: reads quarantine
- Validation command → Dataset validation: runs checks (evidence: `cmd/bq-otel-validate/main.go`)
- Dataset validation → BigQuery tables: checks destination
- Benchmark command → Benchmark runner: runs benchmark (evidence: `cmd/bq-otel-testbench/main.go`)
- Benchmark runner → Signal exporter: creates exporters (evidence: `internal/cli/testbench/runner.go`)
