# datadog/datadog-serverless-functions architecture

> Architecture diagram of the GitHub repository datadog/datadog-serverless-functions, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datadog/datadog-serverless-functions
- Repository: https://github.com/datadog/datadog-serverless-functions

## Overview

This repository contains serverless integrations that process AWS and Azure log events and send normalized data to Datadog; its AWS logs package also includes a trace-forwarding runtime. The main log workflow is event handling, parsing and enrichment, then batching/scrubbing and intake delivery. Retry storage and metadata caches are supporting stateful capabilities. The sampled sources establish several handler behaviors and client calls, but do not expose the AWS Lambda entry-point wiring or all Azure trigger-to-handler calls, so those links are omitted.

Main entry points: [`aws/logs_monitoring/caching/cache_layer.py`](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/cache_layer.py) (Cache layer), [`aws/logs_monitoring/steps/handlers/awslogs_handler.py`](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/awslogs_handler.py) (CloudWatch handler), [`aws/logs_monitoring/steps/handlers/s3_handler.py`](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/s3_handler.py) (S3 handler), [`aws/logs_monitoring/trace_forwarder/cmd/trace/main.go`](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/trace_forwarder/cmd/trace/main.go) (Trace forwarder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_aws_logs["AWS log processing"]
  node_awslogs_handler["CloudWatch handler<br/>[awslogs_handler.py]"]
  node_s3_handler["S3 handler<br/>[s3_handler.py]"]
  node_aws_attributes["AWS attributes<br/>[aws_attributes.py]"]
  node_steps["Log processing steps"]
  node_aws_entry["AWS log forwarder<br/>[lambda_function.py]"]
end

subgraph group_delivery["Datadog delivery"]
  node_batcher["Log batcher<br/>[datadog_batcher.py]"]
  node_datadog_client["Retrying client<br/>[datadog_client.py]"]
  node_http_client["HTTP intake client"]
end

subgraph group_state["Caches and retries"]
  node_cache_layer["Cache layer<br/>[cache_layer.py]"]
  node_cloudwatch_cache["Log-group tags cache"]
  node_s3_tags_cache["S3 tags cache<br/>[s3_tags_cache.py]"]
  node_lambda_cache["Lambda cache<br/>[lambda_cache.py]"]
  node_retry_storage["Retry storage interface<br/>[base_storage.py]"]
  node_s3_retry["S3 retry storage<br/>[storage.py]"]
  node_sqs_retry["SQS retry storage<br/>[sqs_storage.py]"]
end

subgraph group_azure["Azure integrations"]
  node_azure_activity["Activity-log function<br/>[index.js]"]
  node_azure_blobs["Blob-log function<br/>[index.js]"]
end

subgraph group_traces["Trace forwarding"]
  node_trace_entry["Trace forwarder<br/>[main.go]"]
  node_trace_processing["Trace processing"]
  node_trace_connection["Trace intake connection"]
end

node_cloudwatch_event(("CloudWatch event"))
node_s3_event(("S3 event"))
node_retry_s3[("AWS S3 retry objects")]
node_retry_sqs["AWS SQS retry queue"]
node_datadog(("Datadog intake"))
node_azure_events(("Azure log events"))
node_trace_source(("Serialized trace payloads"))
node_trace_intake(("Datadog trace intake"))

node_cloudwatch_event -->|"provides event"| node_awslogs_handler
node_s3_event -->|"provides event"| node_s3_handler
node_awslogs_handler -->|"builds attributes"| node_aws_attributes
node_awslogs_handler -->|"reads tags"| node_cache_layer
node_s3_handler -->|"reads tags"| node_cache_layer
node_awslogs_handler -->|"uses helpers"| node_steps
node_s3_handler -->|"uses helpers"| node_steps
node_aws_entry -.->|"runtime scope"| node_steps
node_cache_layer -->|"accesses"| node_cloudwatch_cache
node_cache_layer -->|"accesses"| node_s3_tags_cache
node_cache_layer -->|"accesses"| node_lambda_cache
node_s3_retry -->|"implements"| node_retry_storage
node_sqs_retry -->|"implements"| node_retry_storage
node_s3_retry -->|"reads and writes"| node_retry_s3
node_sqs_retry -->|"reads and writes"| node_retry_sqs
node_http_client -->|"posts logs"| node_datadog
node_datadog_client -->|"retries send"| node_http_client
node_azure_events -->|"invokes"| node_azure_activity
node_azure_events -->|"invokes"| node_azure_blobs
node_azure_activity -.->|"sends logs"| node_datadog
node_azure_blobs -->|"sends batches"| node_datadog
node_trace_source -->|"provides payload"| node_trace_entry
node_trace_entry -->|"processes traces"| node_trace_processing
node_trace_entry -->|"sends traces"| node_trace_connection
node_trace_connection -->|"forwards traces"| node_trace_intake

click node_awslogs_handler "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/awslogs_handler.py"
click node_s3_handler "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/s3_handler.py"
click node_aws_attributes "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/aws_attributes.py"
click node_steps "https://github.com/datadog/datadog-serverless-functions/tree/master/aws/logs_monitoring/steps"
click node_aws_entry "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/lambda_function.py"
click node_cache_layer "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/cache_layer.py"
click node_cloudwatch_cache "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/cloudwatch_log_group_cache.py"
click node_s3_tags_cache "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/s3_tags_cache.py"
click node_lambda_cache "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/lambda_cache.py"
click node_retry_storage "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/retry/base_storage.py"
click node_s3_retry "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/retry/storage.py"
click node_sqs_retry "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/retry/sqs_storage.py"
click node_batcher "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/logs/datadog_batcher.py"
click node_datadog_client "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/logs/datadog_client.py"
click node_http_client "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/logs/datadog_http_client.py"
click node_azure_activity "https://github.com/datadog/datadog-serverless-functions/blob/master/azure/activity_logs_monitoring/index.js"
click node_azure_blobs "https://github.com/datadog/datadog-serverless-functions/blob/master/azure/blobs_logs_monitoring/index.js"
click node_trace_entry "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/trace_forwarder/cmd/trace/main.go"
click node_trace_processing "https://github.com/datadog/datadog-serverless-functions/tree/master/aws/logs_monitoring/trace_forwarder/internal/apm"
click node_trace_connection "https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/trace_forwarder/internal/apm/trace-connection.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_awslogs_handler,node_s3_handler,node_aws_attributes,node_steps,node_aws_entry toneBlue
class node_batcher,node_datadog_client,node_http_client,node_retry_s3 toneAmber
class node_cache_layer,node_cloudwatch_cache,node_s3_tags_cache,node_lambda_cache,node_retry_storage,node_s3_retry,node_sqs_retry toneMint
class node_azure_activity,node_azure_blobs,node_retry_sqs toneRose
class node_trace_entry,node_trace_processing,node_trace_connection,node_cloudwatch_event,node_s3_event,node_datadog,node_azure_events,node_trace_source,node_trace_intake toneIndigo
```

## Components

### AWS log processing

- [CloudWatch handler](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/awslogs_handler.py): `aws/logs_monitoring/steps/handlers/awslogs_handler.py`
- [S3 handler](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/s3_handler.py): `aws/logs_monitoring/steps/handlers/s3_handler.py`
- [AWS attributes](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/steps/handlers/aws_attributes.py): `aws/logs_monitoring/steps/handlers/aws_attributes.py`
- [Log processing steps](https://github.com/datadog/datadog-serverless-functions/tree/master/aws/logs_monitoring/steps): `aws/logs_monitoring/steps`
- [AWS log forwarder](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/lambda_function.py): `aws/logs_monitoring/lambda_function.py`

### Datadog delivery

- [Log batcher](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/logs/datadog_batcher.py): `aws/logs_monitoring/logs/datadog_batcher.py`
- [Retrying client](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/logs/datadog_client.py): `aws/logs_monitoring/logs/datadog_client.py`
- [HTTP intake client](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/logs/datadog_http_client.py): `aws/logs_monitoring/logs/datadog_http_client.py`

### Caches and retries

- [Cache layer](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/cache_layer.py): `aws/logs_monitoring/caching/cache_layer.py`
- [Log-group tags cache](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/cloudwatch_log_group_cache.py): `aws/logs_monitoring/caching/cloudwatch_log_group_cache.py`
- [S3 tags cache](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/s3_tags_cache.py): `aws/logs_monitoring/caching/s3_tags_cache.py`
- [Lambda cache](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/caching/lambda_cache.py): `aws/logs_monitoring/caching/lambda_cache.py`
- [Retry storage interface](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/retry/base_storage.py): `aws/logs_monitoring/retry/base_storage.py`
- [S3 retry storage](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/retry/storage.py): `aws/logs_monitoring/retry/storage.py`
- [SQS retry storage](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/retry/sqs_storage.py): `aws/logs_monitoring/retry/sqs_storage.py`

### Azure integrations

- [Activity-log function](https://github.com/datadog/datadog-serverless-functions/blob/master/azure/activity_logs_monitoring/index.js): `azure/activity_logs_monitoring/index.js`
- [Blob-log function](https://github.com/datadog/datadog-serverless-functions/blob/master/azure/blobs_logs_monitoring/index.js): `azure/blobs_logs_monitoring/index.js`

### Trace forwarding

- [Trace forwarder](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/trace_forwarder/cmd/trace/main.go): `aws/logs_monitoring/trace_forwarder/cmd/trace/main.go`
- [Trace processing](https://github.com/datadog/datadog-serverless-functions/tree/master/aws/logs_monitoring/trace_forwarder/internal/apm): `aws/logs_monitoring/trace_forwarder/internal/apm`
- [Trace intake connection](https://github.com/datadog/datadog-serverless-functions/blob/master/aws/logs_monitoring/trace_forwarder/internal/apm/trace-connection.go): `aws/logs_monitoring/trace_forwarder/internal/apm/trace-connection.go`

### Other components

- **CloudWatch event**
- **S3 event**
- **AWS S3 retry objects**
- **AWS SQS retry queue**
- **Datadog intake**
- **Azure log events**
- **Serialized trace payloads**
- **Datadog trace intake**

## Connections

- CloudWatch event → CloudWatch handler: provides event
- S3 event → S3 handler: provides event
- CloudWatch handler → AWS attributes: builds attributes
- CloudWatch handler → Cache layer: reads tags
- S3 handler → Cache layer: reads tags
- CloudWatch handler → Log processing steps: uses helpers
- S3 handler → Log processing steps: uses helpers
- AWS log forwarder → Log processing steps: runtime scope
- Cache layer → Log-group tags cache: accesses
- Cache layer → S3 tags cache: accesses
- Cache layer → Lambda cache: accesses
- S3 retry storage → Retry storage interface: implements
- SQS retry storage → Retry storage interface: implements
- S3 retry storage → AWS S3 retry objects: reads and writes
- SQS retry storage → AWS SQS retry queue: reads and writes
- HTTP intake client → Datadog intake: posts logs
- Retrying client → HTTP intake client: retries send
- Azure log events → Activity-log function: invokes
- Azure log events → Blob-log function: invokes
- Activity-log function → Datadog intake: sends logs
- Blob-log function → Datadog intake: sends batches
- Serialized trace payloads → Trace forwarder: provides payload
- Trace forwarder → Trace processing: processes traces
- Trace forwarder → Trace intake connection: sends traces
- Trace intake connection → Datadog trace intake: forwards traces