How datadog/datadog-serverless-functions works
This repository contains serverless integrations that process AWS and Azure log events and send normalized data to Datadog; its AWS logs package also includes a trace-forwarding runtime. The main log workflow is event handling, parsing and enrichment, then batching/scrubbing and intake delivery. Retry storage and metadata caches are supporting stateful capabilities. The sampled sources establish several handler behaviors and client calls, but do not expose the AWS Lambda entry-point wiring or all Azure trigger-to-handler calls, so those links are omitted.
Main entry points: cache_layer.py Cache layer · awslogs_handler.py CloudWatch handler · s3_handler.py S3 handler · main.go Trace forwarder
Components 28
AWS log processing
- CloudWatch handler
aws/logs_monitoring/steps/handlers/awslogs_handler.py - S3 handler
aws/logs_monitoring/steps/handlers/s3_handler.py - AWS attributes
aws/logs_monitoring/steps/handlers/aws_attributes.py - Log processing steps
aws/logs_monitoring/steps - AWS log forwarder
aws/logs_monitoring/lambda_function.py
Datadog delivery
- Log batcher
aws/logs_monitoring/logs/datadog_batcher.py - Retrying client
aws/logs_monitoring/logs/datadog_client.py - HTTP intake client
aws/logs_monitoring/logs/datadog_http_client.py
Caches and retries
- Cache layer
aws/logs_monitoring/caching/cache_layer.py - Log-group tags cache
aws/logs_monitoring/caching/cloudwatch_log_group_cache.py - S3 tags cache
aws/logs_monitoring/caching/s3_tags_cache.py - Lambda cache
aws/logs_monitoring/caching/lambda_cache.py - Retry storage interface
aws/logs_monitoring/retry/base_storage.py - S3 retry storage
aws/logs_monitoring/retry/storage.py - SQS retry storage
aws/logs_monitoring/retry/sqs_storage.py
Azure integrations
- Activity-log function
azure/activity_logs_monitoring/index.js - Blob-log function
azure/blobs_logs_monitoring/index.js
Trace forwarding
- Trace forwarder
aws/logs_monitoring/trace_forwarder/cmd/trace/main.go - Trace processing
aws/logs_monitoring/trace_forwarder/internal/apm - Trace intake connection
aws/logs_monitoring/trace_forwarder/internal/apm/trace-connection.go
Other components
- CloudWatch event
- S3 event
- AWS S3 retry objects
- AWS SQS retry queue
- Datadog intake
- Azure log events
- Serialized trace payloads
- Datadog trace intake
Connections 25
- CloudWatch event → CloudWatch handler: provides event
- S3 event → S3 handler: provides event
- CloudWatch handler → AWS attributes: builds attributes
- CloudWatch handler → Cache layer: reads tags
- S3 handler → Cache layer: reads tags
- CloudWatch handler → Log processing steps: uses helpers
- S3 handler → Log processing steps: uses helpers
- AWS log forwarder → Log processing steps: runtime scope
- Cache layer → Log-group tags cache: accesses
- Cache layer → S3 tags cache: accesses
- Cache layer → Lambda cache: accesses
- S3 retry storage → Retry storage interface: implements
- SQS retry storage → Retry storage interface: implements
- S3 retry storage → AWS S3 retry objects: reads and writes
- SQS retry storage → AWS SQS retry queue: reads and writes
- HTTP intake client → Datadog intake: posts logs
- Retrying client → HTTP intake client: retries send
- Azure log events → Activity-log function: invokes
- Azure log events → Blob-log function: invokes
- Activity-log function → Datadog intake: sends logs
- Blob-log function → Datadog intake: sends batches
- Serialized trace payloads → Trace forwarder: provides payload
- Trace forwarder → Trace processing: processes traces
- Trace forwarder → Trace intake connection: sends traces
- Trace intake connection → Datadog trace intake: forwards traces