How mandark-droid/genai_otel_instrument works
Project Type and Purpose
• This is an open-source Python library (genai-otel-instrument) that provides zero-code and programmatic OpenTelemetry instrumentation for GenAI/LLM applications and related services (databases, caches, message queues, vector DBs).
• It auto-instruments 15+ LLM providers (OpenAI, Anthropic, HuggingFace, etc.) and MCP (“Model Context Protocol”) tools (Postgres, Redis, Kafka, Pinecone, etc.), collects traces & metrics (including cost and GPU usage), enriches spans with cost attributes, and exports them via OTLP to any OpenTelemetry-compatible backend.
Key Architectural Elements from File Structure
• genai_otel/auto_instrument.py & cli.py – entry points for instrumentation (API or CLI wrapper).
• genai_otel/config.py – holds configuration logic (env vars, programmatic).
• genai_otel/instrumentors/*.py – plugin modules that hook into each LLM provider SDK (monkey-patch or decorator pattern).
• genai_otel/mcp_instrumentors/*.py – similar plugin modules for MCP tools (DBs, caches, APIs, vector DBs).
• cost_calculator.py, cost_enrichment_processor.py – logic to compute and enrich spans with cost metrics.
• gpu_metrics.py – collects GPU utilization & power metrics periodically.
• logging_config.py, metrics.py – auxiliary modules.
• example_usage.py & examples/demo – a reference app + Docker-Compose for demo telemetry stack (OTEL Collector, Prometheus, Grafana, OpenSearch).
Information from README
• Zero-code setup via environment variables or one-line API call or CLI wrapper.
• Deep lists of supported providers & MCP tools.
• Exposes spans & metrics with rich attributes per OpenTelemetry semantic conventions (GenAI).
• Configurable exporters, timeouts, semantic conventions, session/user extraction, RAG pipeline helpers.
a. Main Components
• User Application (any Python app using LLMs, DBs, caches, vector DBs)
• genai-otel Instrumentation Library
– Entry Point: genai_otel.instrument() or genai-instrument CLI
– Config Manager (reads env vars / programmatic)
– Instrumentor Manager (discovers & registers instrumentors)
– Instrumentors
· LLM Instrumentors (openai, anthropic, huggingface, etc.)
· MCP Instrumentors (Postgres, Redis, Kafka, Pinecone, etc.)
– Cost Calculator & Enrichment Processor
– GPU Metrics Collector
• OpenTelemetry SDK Pipeline
– Span & Metric Processors (CostEnrichmentSpanProcessor, BatchSpanProcessor)
– OTLP Exporter
• External Dependencies
– LLM Provider APIs (OpenAI, Anthropic, Google AI, etc.)
– MCP Services (Postgres, Redis, Kafka, Vector DBs)
• Observability Backends
– OTEL Collector / Direct OTLP Endpoint
– Tracing Backends (Jaeger, Zipkin, Elastic APM, etc.)
– Metrics Backends (Prometheus + Grafana, OpenSearch + Kibana, Datadog, etc.)
b. Relationships & Data Flows
1. User App calls LLM/DB client → Instrumentor intercepts and starts a span (attributes: model, tokens, SQL, etc.)
2. Instrumentor calls original SDK function → receives response/result
3. Instrumentor captures response attributes (latency, token counts), computes cost via Cost Calculator
4. Span & metrics (counters, histograms, gauges) emitted to OpenTelemetry SDK
5. SpanProcessor enriches spans with cost and GPU attributes, batches them
6. OTLP Exporter transmits telemetry to configured endpoint
7. Telemetry backend stores & visualizes traces/metrics (e.g., Grafana dashboards in examples/demo)
c. Architectural Patterns & Principles
• Plugin/Instrumentor Pattern: each provider or service has its own module implementing a standard interface (monkey-patch or decorator).
• OpenTelemetry Pipeline: separation of concern between instrumentation, processing, and exporting.
• Zero-Code vs. Code-Based API: entry points enable both environment-driven and API-driven configurations.
• Config-Driven Behavior: uses environment variables and a unified config class for feature toggles (GPU, cost tracking, MCP).
d. Key Technologies & Frameworks
• Python 3.9+
• OpenTelemetry SDK (tracing & metrics)
• OTLP exporter (HTTP/Protobuf or gRPC)
• LLM SDKs (openai, anthropic, cohere, etc.)
• DB/Cache clients (psycopg2, redis-py, kafka-python, pinecone, etc.)
• Prometheus, Grafana, OpenSearch for demos