# insightfinder/insightagent architecture

> Architecture diagram of the GitHub repository insightfinder/insightagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/insightfinder/insightagent
- Repository: https://github.com/insightfinder/insightagent

## Overview

##### Project type and purpose
- This is a monolithic “monorepo” containing an open-source collection of monitoring agents plus a handful of supporting microservices.
- Its core purpose is to collect metrics and logs from a wide variety of sources (OS, containers, cloud services, databases, message buses, etc.) and forward them into the InsightFinder platform (via HTTP APIs, Kafka, or other channels).

##### Key areas in the file tree
● common/
– Shared Python modules for configuration, logging, and pushing data (reportMetrics.py, reportLog.py, ifLogger.py, etc.)
● deployment/
– Ansible playbooks, shell scripts, cron-configs to install/configure agents on hosts
● [agent folders] (hundreds of per-source subdirectories)
– Python scripts (getmetrics_*.py, getlogs_*.py), Go programs (main.go), Java Maven apps, Dockerfiles, config templates, offline pip wheels
– Each folder follows the same pattern: config.ini.template → language-specific “collector” → uses common/ utilities → runs under cron or Docker
● KafkaCollectorAgent/
– Java/Spring Boot service that consumes from Kafka topics and forwards metric/log messages into InsightFinder via REST
● grafana-agent/, kubeactions/, saml/, webhook/
– Standalone microservices (Go or Java) that integrate with external systems (Grafana API, Kubernetes webhook admission, SAML IdP, Helm-deployed webhooks)
● TeamsTestingUtil/
– A utility app (Java) for testing notifications in Microsoft Teams

##### Major components
- Agent Runtimes (grouped by language):
- Python-based agents (cron + Python scripts)
- Go-based agents (compiled binaries, Docker containers)
- Java-based agents (Maven, Spring Boot)
- Shared library (“common” Python module)
- Deployment layer: Ansible roles, shell scripts, Docker deployments
- Collector services:
- KafkaCollectorAgent (Java)
- Grafana-agent (Go)
- Integration microservices:
- KubeActions service (Java Spring)
- SAML auth service (Java Spring)
- Webhook service (Go + Helm chart)
- Downstream platform: InsightFinder ingestion API (external HTTP endpoints)

##### Relationships and data flows
1. Agents poll or subscribe to their respective data sources (OS stats, cloud APIs, DBs, message queues).
2. Collected data flows through common/ utilities (parsing, batching, serialization).
3. Data is pushed via:
- Direct HTTP REST calls to InsightFinder ingestion endpoints
- Kafka topics (for high-scale sources), then consumed by KafkaCollectorAgent → HTTP → InsightFinder
4. Deployment/Ansible ensures each agent’s config, pip packages or Docker image, and monit/cron are installed on the host.
5. Grafana-agent fetches dashboards or metrics from Grafana REST API → forwards to InsightFinder.
6. KubeActions microservice opens WebSocket to InsightFinder and Kubernetes API server; applies scaling instructions back to the cluster.
7. SAML service provides authentication for the InsightFinder UI; Webhook service receives external events (via HTTP) → forwards into InsightFinder.

##### Architectural patterns and principles
- Micro-service style: many small, single-purpose collectors
- Shared library for cross-cutting concerns (common/)
- Configuration-as-code: templated ini/yaml under each agent + Ansible roles
- Polyglot: Python, Go, Java Lego-brick approach
- Event-driven pipeline: cron → agent → queue or HTTP → ingestion
- Containerization for isolation: many Dockerfiles for agents & services

##### Essential callouts
- KafkaCollectorAgent decouples high-throughput sources from the HTTP ingestion.
- Deployment via Ansible and Docker ensures consistent rollout across diverse environments.
- Microservices integrate with third-party APIs and the InsightFinder control plane (WebSocket, SAML, webhook).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Deployment Pipeline"
        deployment[(Ansible & Scripts)]:::deployment
        click deployment "https://github.com/insightfinder/insightagent/tree/master/deployment/"
    end

    subgraph "Data Sources"
        OS["OS / Containers"]:::external
        Cloud["Cloud Services"]:::external
        DB["Databases"]:::external
        MQ["Message Queues"]:::external
    end

    subgraph "Agent Runtimes"
        subgraph "Python Agents"
            py1["aws-s3-metadata"]:::python
            click py1 "https://github.com/insightfinder/insightagent/tree/master/aws-s3-metadata/"
        end
        subgraph "Go Agents"
            go1["dell_power_collector"]:::go
            click go1 "https://github.com/insightfinder/insightagent/tree/master/dell_power_collector/"
            go2["kubernetes-agent"]:::go
            click go2 "https://github.com/insightfinder/insightagent/tree/master/kubernetes-agent/"
            go3["prometheus-go"]:::go
            click go3 "https://github.com/insightfinder/insightagent/tree/master/prometheus-go/"
        end
        subgraph "Java Agents"
            ja1["influxDBAgent"]:::java
            click ja1 "https://github.com/insightfinder/insightagent/tree/master/influxDBAgent/"
        end
    end

    commonLib["common"]:::shared
    click commonLib "https://github.com/insightfinder/insightagent/tree/master/common/"

    OS --> py1
    Cloud --> py1
    DB --> py1
    MQ --> py1

    OS --> go1
    Cloud --> go1
    DB --> go1
    MQ --> go1

    OS --> go2
    Cloud --> go2
    DB --> go2
    MQ --> go2

    OS --> go3
    Cloud --> go3
    DB --> go3
    MQ --> go3

    OS --> ja1
    Cloud --> ja1
    DB --> ja1
    MQ --> ja1

    py1 --> commonLib
    go1 --> commonLib
    go2 --> commonLib
    go3 --> commonLib
    ja1 --> commonLib

    commonLib -->|"HTTP REST"| IF["InsightFinder API"]:::external
    commonLib -->|"Kafka"| KafkaBroker["Kafka Broker"]:::external

    KafkaBroker --> KafkaAgent["KafkaCollectorAgent"]:::java
    click KafkaAgent "https://github.com/insightfinder/insightagent/tree/master/KafkaCollectorAgent/"
    KafkaAgent -->|"HTTP REST"| IF

    subgraph "Integration Services"
        graf["grafana-agent"]:::go
        click graf "https://github.com/insightfinder/insightagent/tree/master/grafana-agent/"
        kube["kubeactions"]:::java
        click kube "https://github.com/insightfinder/insightagent/tree/master/kubeactions/"
        samlS["saml"]:::java
        click samlS "https://github.com/insightfinder/insightagent/tree/master/saml/"
        webhookS["webhook"]:::go
        click webhookS "https://github.com/insightfinder/insightagent/tree/master/webhook/"
        teams["TeamsTestingUtil"]:::java
        click teams "https://github.com/insightfinder/insightagent/tree/master/TeamsTestingUtil/"
    end

    graf -->|"HTTP REST"| IF
    webhookS -->|"HTTP REST"| IF
    samlS -->|"Auth for UI"| IFUI["InsightFinder UI"]:::external
    teams -->|"Test Notifications"| IF

    IFWS["WebSocket Endpoint"]:::external
    IFWS -->|WebSocket| kube
    kube -->|"K8s API Calls"| K8sAPI["Kubernetes API"]:::external

    deployment --> py1
    deployment --> go1
    deployment --> go2
    deployment --> go3
    deployment --> ja1
    deployment --> graf
    deployment --> kube
    deployment --> samlS
    deployment --> webhookS

    classDef python fill:#c6f7c3,stroke:#2d8f2d,color:#000;
    classDef go fill:#c3d7f0,stroke:#11457e,color:#000;
    classDef java fill:#f7d7c3,stroke:#7e460f,color:#000;
    classDef shared fill:#ebebeb,stroke:#888888,color:#000;
    classDef deployment fill:#fff8b3,stroke:#e3b505,color:#000;
    classDef external fill:#f0f0f0,stroke:#999999,color:#000;
```
