# ductan2/elk-repo architecture

> Architecture diagram of the GitHub repository ductan2/elk-repo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ductan2/elk-repo
- Repository: https://github.com/ductan2/elk-repo

## Overview

##### Main components
- Docker Compose Orchestration: Represent the docker-compose.yml file, which is used to spin up and manage the containers.
- Application Component (app): This directory contains a Dockerfile, a Python script (main.py) and its dependencies (requirements.txt). It likely represents a custom app that integrates with or simulates production logs/transactions.
- Beats: Files such as filebeat.yml and metricbeat.yml and the config_examples directory (with sample filebeat and kibana config files) indicate that Filebeat (and possibly Metricbeat) are used to collect and forward log and metric data.
- Logstash: With a logstash.conf file and sample data in the logstash_ingest_data directory, Logstash is configured to ingest CSV data (e.g., Air_Quality.csv) and process logs.
- Kibana: The presence of kibana.yml and its sample in config_examples shows that Kibana is part of the stack for visualization and dashboards.
- Ingest Data Directories: filebeat_ingest_data and logstash_ingest_data contain sample log data (Air_Quality.log and Air_Quality.csv) that serve to simulate production usage or test pipeline flows.
- Additional Utility Scripts: get-cert-details.sh suggests some support for certificate or security-related tasks.

##### Relationships and interactions
- The docker-compose file ties together all the services and ensures that the application, Beats (Filebeat/Metricbeat), Logstash, Elasticsearch, and Kibana are networked appropriately.
- The “app” service likely generates logs or APM telemetry, which then flows into Beats (Filebeat) for log shipping.
- Filebeat collects logs (using its configuration files) from the designated directories (or from the application) and forwards data to either Logstash or directly to Elasticsearch.
- Logstash, configured by logstash.conf, processes the incoming CSV/log data (e.g., from the logstash_ingest_data directory) before sending it on to Elasticsearch.
- Kibana (configured via kibana.yml) interacts with Elasticsearch to provide visual dashboards.
- The configuration examples and environment settings (.env) ensure that all components are properly parameterized and can interact seamlessly.

##### Architectural patterns and design principles
- Containerization and Orchestration: The use of Docker Compose emphasizes containerization, which isolates components (app, Beats, Logstash, Kibana) and defines clear service boundaries.
- Decoupling and Modularization: Logs/data ingestion is modular (separate directories, configuration examples) and can be extended or reconfigured independently.
- Pipeline Architecture: The flow from log/metric generation (app) through forwarding agents (Beats) to processing (Logstash) and finally visualization (Kibana) demonstrates a typical ETL (Extract, Transform, Load) pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    DC["Docker Compose Orchestration (docker-compose.yml)"]:::container
    App["Application Service (Python, Dockerfile, main.py)"]:::container
    
    subgraph "Beats"
        FB["Filebeat (Log Collector)"]:::container
        MB["Metricbeat (Metrics)"]:::container
    end

    LS["Logstash (Data Processor - logstash.conf)"]:::container
    ES["Elasticsearch (Data Storage)"]:::storage
    KB["Kibana (Visualization - kibana.yml)"]:::container

    FID["Filebeat Ingest Data (filebeat_ingest_data/)"]:::config
    LID["Logstash Ingest Data (logstash_ingest_data/)"]:::config
    UT["Utility Script (get-cert-details.sh)"]:::utility

    %% Docker Compose orchestration connects all containerized services
    DC --> App
    DC --> FB
    DC --> MB
    DC --> LS
    DC --> KB
    DC --> UT

    %% Data Flow connections
    App -->|"generates_logs/APM"| FB
    FB -->|"forwards_logs"| LS
    LS -->|"processes_data"| ES
    ES -->|"feeds_data"| KB

    %% Ingest data connections
    FB --- FID
    LS --- LID

    %% Click Events
    click DC "https://github.com/ductan2/elk-repo/blob/main/docker-compose.yml"
    click App "https://github.com/ductan2/elk-repo/tree/main/app/"
    click FB "https://github.com/ductan2/elk-repo/blob/main/filebeat.yml"
    click MB "https://github.com/ductan2/elk-repo/blob/main/metricbeat.yml"
    click LS "https://github.com/ductan2/elk-repo/blob/main/logstash.conf"
    click LID "https://github.com/ductan2/elk-repo/tree/main/logstash_ingest_data/"
    click KB "https://github.com/ductan2/elk-repo/blob/main/kibana.yml"
    click FID "https://github.com/ductan2/elk-repo/tree/main/filebeat_ingest_data/"
    click UT "https://github.com/ductan2/elk-repo/blob/main/get-cert-details.sh"

    %% Styles
    classDef container fill:#a2d9ff,stroke:#004080,stroke-width:2px;
    classDef storage fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef config fill:#c8f7c5,stroke:#2d6a4f,stroke-width:2px;
    classDef utility fill:#f5b7b1,stroke:#943126,stroke-width:2px;
```
