# aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service architecture

> Architecture diagram of the GitHub repository aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service
- Repository: https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service

## Overview

This CDK application provisions an event-driven Apache access-log ingestion and search solution. Logs uploaded to S3 generate SQS notifications; an OpenSearch Ingestion pipeline reads the S3 objects, parses and enriches Apache records, then writes monthly indices to an OpenSearch domain. The CDK entry point composes the storage, IAM, domain, and logging resources. README also documents creating an index template and importing a saved dashboard, but those are manual console steps rather than implemented runtime components here.

Main entry points: [`app.py`](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/app.py) (CDK app), [`cdk/pipeline_stack.py`](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/pipeline_stack.py) (Ingestion pipeline), [`cdk/ingestion_stack.py`](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py) (Log bucket), [`cdk/opensearch_stack.py`](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/opensearch_stack.py) (OpenSearch domain)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_provision["CDK composition"]
  node_cdk_app["CDK app<br/>[app.py]"]
end

subgraph group_ingest["Log ingestion"]
  node_log_bucket[("Log bucket<br/>[ingestion_stack.py]")]
  node_event_queue["Event queue<br/>[ingestion_stack.py]"]
  node_pipeline_role["Pipeline role<br/>[ingestion_stack.py]"]
  node_osis_pipeline["Ingestion pipeline<br/>[pipeline_stack.py]"]
  node_apache_logs["Apache logs"]
end

subgraph group_search["Search and observability"]
  node_opensearch_domain[("OpenSearch domain")]
  node_cloudwatch_groups[("CloudWatch log groups")]
  node_index_template["Index template<br/>[index_template.txt]"]
  node_dashboard["Saved dashboard"]
end

node_operator(("Operator"))
node_dashboard_user(("Dashboard user"))

node_operator -->|"deploys"| node_cdk_app
node_cdk_app -->|"creates"| node_cloudwatch_groups
node_cdk_app -->|"creates"| node_log_bucket
node_cdk_app -->|"creates"| node_opensearch_domain
node_cdk_app -->|"creates"| node_osis_pipeline
node_log_bucket -->|"notifies on writes"| node_event_queue
node_pipeline_role -->|"consumes messages"| node_event_queue
node_osis_pipeline -->|"polls events"| node_event_queue
node_osis_pipeline -->|"reads objects"| node_log_bucket
node_osis_pipeline -->|"parses and indexes"| node_opensearch_domain
node_opensearch_domain -->|"publishes logs"| node_cloudwatch_groups
node_osis_pipeline -->|"publishes logs"| node_cloudwatch_groups
node_operator -->|"uploads logs"| node_log_bucket
node_apache_logs -->|"uploaded to"| node_log_bucket
node_dashboard_user -->|"views ingested data"| node_opensearch_domain
node_dashboard_user -->|"creates manually"| node_index_template
node_dashboard_user -->|"imports manually"| node_dashboard

click node_cdk_app "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/app.py"
click node_log_bucket "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py"
click node_event_queue "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py"
click node_pipeline_role "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py"
click node_osis_pipeline "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/pipeline_stack.py"
click node_apache_logs "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/apache_log_data/apache_access_log.gz"
click node_opensearch_domain "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/opensearch_stack.py"
click node_cloudwatch_groups "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/cloudwatch_stack.py"
click node_index_template "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/opensearch_objects/index_template.txt"
click node_dashboard "https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/opensearch_objects/apache_access_log_dashboard.ndjson"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cdk_app,node_dashboard_user toneBlue
class node_log_bucket,node_event_queue,node_pipeline_role,node_osis_pipeline,node_apache_logs toneAmber
class node_opensearch_domain,node_cloudwatch_groups,node_index_template,node_dashboard toneMint
class node_operator toneIndigo
```

## Components

### CDK composition

- [CDK app](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/app.py): `app.py`

### Log ingestion

- [Log bucket](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py): `cdk/ingestion_stack.py`
- [Event queue](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py): `cdk/ingestion_stack.py`
- [Pipeline role](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/ingestion_stack.py): `cdk/ingestion_stack.py`
- [Ingestion pipeline](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/pipeline_stack.py): `cdk/pipeline_stack.py`
- [Apache logs](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/apache_log_data/apache_access_log.gz): `apache_log_data/apache_access_log.gz`

### Search and observability

- [OpenSearch domain](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/opensearch_stack.py): `cdk/opensearch_stack.py`
- [CloudWatch log groups](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/cdk/cloudwatch_stack.py): `cdk/cloudwatch_stack.py`
- [Index template](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/opensearch_objects/index_template.txt): `opensearch_objects/index_template.txt`
- [Saved dashboard](https://github.com/aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service/blob/main/opensearch_objects/apache_access_log_dashboard.ndjson): `opensearch_objects/apache_access_log_dashboard.ndjson`

### Other components

- **Operator**
- **Dashboard user**

## Connections

- Operator → CDK app: deploys (evidence: `README.md`)
- CDK app → CloudWatch log groups: creates (evidence: `app.py`)
- CDK app → Log bucket: creates (evidence: `app.py`)
- CDK app → OpenSearch domain: creates (evidence: `app.py`)
- CDK app → Ingestion pipeline: creates (evidence: `app.py`)
- Log bucket → Event queue: notifies on writes (evidence: `cdk/ingestion_stack.py`)
- Pipeline role → Event queue: consumes messages (evidence: `cdk/ingestion_stack.py`)
- Ingestion pipeline → Event queue: polls events (evidence: `cdk/pipeline_stack.py`)
- Ingestion pipeline → Log bucket: reads objects (evidence: `cdk/ingestion_stack.py`)
- Ingestion pipeline → OpenSearch domain: parses and indexes (evidence: `cdk/pipeline_stack.py`)
- OpenSearch domain → CloudWatch log groups: publishes logs (evidence: `cdk/opensearch_stack.py`)
- Ingestion pipeline → CloudWatch log groups: publishes logs (evidence: `cdk/pipeline_stack.py`)
- Operator → Log bucket: uploads logs (evidence: `README.md`)
- Apache logs → Log bucket: uploaded to (evidence: `README.md`)
- Dashboard user → OpenSearch domain: views ingested data (evidence: `README.md`)
- Dashboard user → Index template: creates manually (evidence: `README.md`)
- Dashboard user → Saved dashboard: imports manually (evidence: `README.md`)
