How ductan2/elk-repo works
Main components
• Docker Compose Orchestration: Represent the docker-compose.yml file, which is used to spin up and manage the containers.
• Application Component (app): This directory contains a Dockerfile, a Python script (main.py) and its dependencies (requirements.txt). It likely represents a custom app that integrates with or simulates production logs/transactions.
• Beats: Files such as filebeat.yml and metricbeat.yml and the config_examples directory (with sample filebeat and kibana config files) indicate that Filebeat (and possibly Metricbeat) are used to collect and forward log and metric data.
• Logstash: With a logstash.conf file and sample data in the logstash_ingest_data directory, Logstash is configured to ingest CSV data (e.g., Air_Quality.csv) and process logs.
• Kibana: The presence of kibana.yml and its sample in config_examples shows that Kibana is part of the stack for visualization and dashboards.
• Ingest Data Directories: filebeat_ingest_data and logstash_ingest_data contain sample log data (Air_Quality.log and Air_Quality.csv) that serve to simulate production usage or test pipeline flows.
• Additional Utility Scripts: get-cert-details.sh suggests some support for certificate or security-related tasks.
Relationships and interactions
• The docker-compose file ties together all the services and ensures that the application, Beats (Filebeat/Metricbeat), Logstash, Elasticsearch, and Kibana are networked appropriately.
• The “app” service likely generates logs or APM telemetry, which then flows into Beats (Filebeat) for log shipping.
• Filebeat collects logs (using its configuration files) from the designated directories (or from the application) and forwards data to either Logstash or directly to Elasticsearch.
• Logstash, configured by logstash.conf, processes the incoming CSV/log data (e.g., from the logstash_ingest_data directory) before sending it on to Elasticsearch.
• Kibana (configured via kibana.yml) interacts with Elasticsearch to provide visual dashboards.
• The configuration examples and environment settings (.env) ensure that all components are properly parameterized and can interact seamlessly.
Architectural patterns and design principles
• Containerization and Orchestration: The use of Docker Compose emphasizes containerization, which isolates components (app, Beats, Logstash, Kibana) and defines clear service boundaries.
• Decoupling and Modularization: Logs/data ingestion is modular (separate directories, configuration examples) and can be extended or reconfigured independently.
• Pipeline Architecture: The flow from log/metric generation (app) through forwarding agents (Beats) to processing (Logstash) and finally visualization (Kibana) demonstrates a typical ETL (Extract, Transform, Load) pipeline.