How aws-samples/sample-log-ingestion-pipeline-for-amazon-opensearch-service works
This CDK application provisions an event-driven Apache access-log ingestion and search solution. Logs uploaded to S3 generate SQS notifications; an OpenSearch Ingestion pipeline reads the S3 objects, parses and enriches Apache records, then writes monthly indices to an OpenSearch domain. The CDK entry point composes the storage, IAM, domain, and logging resources. README also documents creating an index template and importing a saved dashboard, but those are manual console steps rather than implemented runtime components here.
Main entry points: app.py CDK app · pipeline_stack.py Ingestion pipeline · ingestion_stack.py Log bucket · opensearch_stack.py OpenSearch domain
Components 12
CDK composition
- CDK app
app.py
Log ingestion
- Log bucket
cdk/ingestion_stack.py - Event queue
cdk/ingestion_stack.py - Pipeline role
cdk/ingestion_stack.py - Ingestion pipeline
cdk/pipeline_stack.py - Apache logs
apache_log_data/apache_access_log.gz
Search and observability
- OpenSearch domain
cdk/opensearch_stack.py - CloudWatch log groups
cdk/cloudwatch_stack.py - Index template
opensearch_objects/index_template.txt - Saved dashboard
opensearch_objects/apache_access_log_dashboard.ndjson
Other components
- Operator
- Dashboard user
Connections 17
- Operator → CDK app: deploys
README.md - CDK app → CloudWatch log groups: creates
app.py - CDK app → Log bucket: creates
app.py - CDK app → OpenSearch domain: creates
app.py - CDK app → Ingestion pipeline: creates
app.py - Log bucket → Event queue: notifies on writes
cdk/ingestion_stack.py - Pipeline role → Event queue: consumes messages
cdk/ingestion_stack.py - Ingestion pipeline → Event queue: polls events
cdk/pipeline_stack.py - Ingestion pipeline → Log bucket: reads objects
cdk/ingestion_stack.py - Ingestion pipeline → OpenSearch domain: parses and indexes
cdk/pipeline_stack.py - OpenSearch domain → CloudWatch log groups: publishes logs
cdk/opensearch_stack.py - Ingestion pipeline → CloudWatch log groups: publishes logs
cdk/pipeline_stack.py - Operator → Log bucket: uploads logs
README.md - Apache logs → Log bucket: uploaded to
README.md - Dashboard user → OpenSearch domain: views ingested data
README.md - Dashboard user → Index template: creates manually
README.md - Dashboard user → Saved dashboard: imports manually
README.md