How googlecloudplatform/bq-otel-exporter works
This repository is an OpenTelemetry Collector exporter that transports traces, logs, and metrics into BigQuery, with optional payload offload and quarantine. Its main path is Collector input → exporter processing and encoding → BigQuery append. Separate commands support archive backfill, quarantine replay, destination validation, and load benchmarking. The README documents downstream views and alerting, but those systems are outside this source tree. Relationships for unsampled internals are included only when supported by the source index or README.
Main entry points: exporter.go Signal exporter · backfill.go Archive backfill · offload.go Payload offload · replay.go Quarantine replay
Components 22
Export runtime
- Signal exporter
exporter.go - Signal mappings
exporter_traces.go - Log mappings
exporter_logs.go - Metric mappings
exporter_metrics.go
Record processing
- Row transforms
internal/transform/rows.go - Metric transforms
internal/transform/metrics.go - Wire encoding
internal/protoenc/wire_encoder.go - Payload offload
internal/offload/offload.go - Redaction
internal/redact/redact.go
Operational commands
- Backfill command
cmd/bq-otel-backfill/main.go - Archive backfill
internal/cli/backfill/backfill.go - Replay command
cmd/bq-quarantine-replay/main.go - Quarantine replay
internal/cli/quarantinereplay/replay.go - Validation command
cmd/bq-otel-validate/main.go - Dataset validation
internal/cli/validate/validate.go - Benchmark command
cmd/bq-otel-testbench/main.go - Benchmark runner
internal/cli/testbench/runner.go
BigQuery destination
- Storage writer
internal/bigquery/writer.go - BigQuery tables
Other components
- Collector
- Cloud Storage
- Views and alerts
Connections 22
- Collector → Signal exporter: sends OTLP
README.md - Signal exporter → Signal mappings: maps traces
exporter.go - Signal exporter → Log mappings: maps logs
exporter.go - Signal exporter → Metric mappings: maps metrics
exporter.go - Signal exporter → Row transforms: transforms records
exporter.go - Signal exporter → Metric transforms: transforms metrics
exporter.go - Signal exporter → Wire encoding: encodes rows
exporter.go - Signal exporter → Redaction: configures redaction
exporter.go - Signal exporter → Payload offload: configures offload
exporter.go - Signal exporter → Storage writer: writes batches
exporter.go - Storage writer → BigQuery tables: appends rows
README.md - Payload offload → Cloud Storage: stores payloads
internal/offload/offload.go - BigQuery tables → Views and alerts: feeds views
README.md - Backfill command → Archive backfill: runs backfill
cmd/bq-otel-backfill/main.go - Archive backfill → BigQuery tables: appends history
- Cloud Storage → Archive backfill: provides archives
- Replay command → Quarantine replay: runs replay
cmd/bq-quarantine-replay/main.go - Quarantine replay → BigQuery tables: reads quarantine
- Validation command → Dataset validation: runs checks
cmd/bq-otel-validate/main.go - Dataset validation → BigQuery tables: checks destination
- Benchmark command → Benchmark runner: runs benchmark
cmd/bq-otel-testbench/main.go - Benchmark runner → Signal exporter: creates exporters
internal/cli/testbench/runner.go