How insightfinder/insightagent works
Project type and purpose
• This is a monolithic “monorepo” containing an open-source collection of monitoring agents plus a handful of supporting microservices.
• Its core purpose is to collect metrics and logs from a wide variety of sources (OS, containers, cloud services, databases, message buses, etc.) and forward them into the InsightFinder platform (via HTTP APIs, Kafka, or other channels).
Key areas in the file tree
● common/
– Shared Python modules for configuration, logging, and pushing data (reportMetrics.py, reportLog.py, ifLogger.py, etc.)
● deployment/
– Ansible playbooks, shell scripts, cron-configs to install/configure agents on hosts
● [agent folders] (hundreds of per-source subdirectories)
– Python scripts (getmetrics_*.py, getlogs_*.py), Go programs (main.go), Java Maven apps, Dockerfiles, config templates, offline pip wheels
– Each folder follows the same pattern: config.ini.template → language-specific “collector” → uses common/ utilities → runs under cron or Docker
● KafkaCollectorAgent/
– Java/Spring Boot service that consumes from Kafka topics and forwards metric/log messages into InsightFinder via REST
● grafana-agent/, kubeactions/, saml/, webhook/
– Standalone microservices (Go or Java) that integrate with external systems (Grafana API, Kubernetes webhook admission, SAML IdP, Helm-deployed webhooks)
● TeamsTestingUtil/
– A utility app (Java) for testing notifications in Microsoft Teams
Major components
• Agent Runtimes (grouped by language):
• Python-based agents (cron + Python scripts)
• Go-based agents (compiled binaries, Docker containers)
• Java-based agents (Maven, Spring Boot)
• Shared library (“common” Python module)
• Deployment layer: Ansible roles, shell scripts, Docker deployments
• Collector services:
• KafkaCollectorAgent (Java)
• Grafana-agent (Go)
• Integration microservices:
• KubeActions service (Java Spring)
• SAML auth service (Java Spring)
• Webhook service (Go + Helm chart)
• Downstream platform: InsightFinder ingestion API (external HTTP endpoints)
Relationships and data flows
1. Agents poll or subscribe to their respective data sources (OS stats, cloud APIs, DBs, message queues).
2. Collected data flows through common/ utilities (parsing, batching, serialization).
3. Data is pushed via:
• Direct HTTP REST calls to InsightFinder ingestion endpoints
• Kafka topics (for high-scale sources), then consumed by KafkaCollectorAgent → HTTP → InsightFinder
4. Deployment/Ansible ensures each agent’s config, pip packages or Docker image, and monit/cron are installed on the host.
5. Grafana-agent fetches dashboards or metrics from Grafana REST API → forwards to InsightFinder.
6. KubeActions microservice opens WebSocket to InsightFinder and Kubernetes API server; applies scaling instructions back to the cluster.
7. SAML service provides authentication for the InsightFinder UI; Webhook service receives external events (via HTTP) → forwards into InsightFinder.
Architectural patterns and principles
• Micro-service style: many small, single-purpose collectors
• Shared library for cross-cutting concerns (common/)
• Configuration-as-code: templated ini/yaml under each agent + Ansible roles
• Polyglot: Python, Go, Java Lego-brick approach
• Event-driven pipeline: cron → agent → queue or HTTP → ingestion
• Containerization for isolation: many Dockerfiles for agents & services
Essential callouts
• KafkaCollectorAgent decouples high-throughput sources from the HTTP ingestion.
• Deployment via Ansible and Docker ensures consistent rollout across diverse environments.
• Microservices integrate with third-party APIs and the InsightFinder control plane (WebSocket, SAML, webhook).