Skip to main content
Version: 3.9.1

Setting up Data source


This guide provides sample scrape job to be used for Prometheus deployment’s scrape-configmap and service monitors to be used with Prometheus operator for the different architectural topologies for integrating Prometheus (connecting a data source link) with Chaos center.

Topologies

Listed below are three among many topologies in which a data source can be setup for collecting chaos delegate cluster's metrics along with chaos metrics for chaos center.

For Control plane chaos delegate / Self chaos delegate with Prometheus - scraping chaos-exporter metrics and events along with other metrics

Data flow and topological diagram

Scrape jobs:

  • Chaos exporter
- job_name: 'chaos-exporter'
static_configs:
- targets: ['chaos-exporter.litmus.svc.cluster.local:8080']
relabel_configs:
- target_label: instance
replacement: 'chaos-exporter-service'
  • Kube state metrics exporter
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']

Detailed setup guide for Prometheus deployment with scrape configuration

Service monitors:

  • Chaos exporter
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: chaos-exporter
labels:
k8s-app: chaos-exporter
namespace: litmus
spec:
jobLabel: app
selector:
matchLabels:
app: chaos-exporter
namespaceSelector:
matchNames:
- litmus
endpoints:
- port: tcp
interval: 1s
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
  • Kube state metrics exporter
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: latest
k8s-app: kube-state-metrics
name: kube-state-metrics
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
honorLabels: true
interval: 30s
port: http-metrics
relabelings:
- action: labeldrop
regex: (pod|service|endpoint|namespace)
scheme: http
scrapeTimeout: 30s
tlsConfig:
insecureSkipVerify: true
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: telemetry
scheme: http
tlsConfig:
insecureSkipVerify: true
jobLabel: app.kubernetes.io/name
selector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics

Detailed setup guide for Prometheus operator with service monitors

Note:

Chaos exporter

labels

labels:
app: chaos-exporter
release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: chaos-exporter-monitor
namespace: monitoring
labels:
release: prometheus-stack
spec:
selector:
matchLabels:
app: chaos-exporter
namespaceSelector:
matchNames:
- litmus
podMetricsEndpoints:
- port: tcp
- interval: 1s
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'

Black box exporter

labels

labels:
app: prometheus-blackbox-exporter
release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: black-box-exporter-monitor
namespace: monitoring
labels:
release: prometheus-stack
spec:
selector:
matchLabels:
app: prometheus-blackbox-exporter
namespaceSelector:
matchNames:
- monitoring
podMetricsEndpoints:
- port: http
- interval: 1s

Detailed setup guide for Prometheus operator with pod monitors

For Multiple chaos delegate s with multiple prometheus instances - scraping chaos-exporter metrics and events along with other metrics

Data flow and topological diagram
  • Scrape job and service monitor remain same as in case of Control plane chaos delegate / self chaos delegate , the individual Prometheus instances can be connected as separate data sources to the Chaos center. Separate dashboards can be created by selecting specific chaos delegate s and their corresponding data source which is essentially a Prometheus time series database, collecting metrics from the chaos delegate cluster for application / infra metrics, chaos events and chaos verdicts.

For Multiple chaos delegate s with single prometheus - scraping chaos-exporter metrics and events along with other metrics

  • Separate dashboards can be created by selecting specific chaos delegate s and the data source which is essentially a Prometheus time series database, collecting metrics from the chaos delegate cluster for application / infra metrics, chaos events and chaos verdicts.
Data flow and topological diagram

Scrape jobs:

  • Chaos-exporter - chaos delegate -1
- job_name: 'chaos-exporter-agent-1'
static_configs:
- targets: ['<AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
relabel_configs:
- target_label: instance
replacement: 'chaos-exporter-service'
  • Chaos-exporter - chaos delegate -2
- job_name: 'chaos-exporter-agent-2'
static_configs:
- targets: ['<AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
relabel_configs:
- target_label: instance
replacement: 'chaos-exporter-service'
  • Kube state metrics exporter - chaos delegate -1
- job_name: 'kube-state-metrics-agent-1'
static_configs:
- targets: ['<AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
  • Kube state metrics exporter - chaos delegate -2
- job_name: 'kube-state-metrics-agent-2'
static_configs:
- targets: ['<AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']

Detailed setup guide for Prometheus deployment with scrape configuration

Service endpoint, spec and monitors:

  • Chaos-exporter - chaos delegate -1
kind: Service
apiVersion: v1
metadata:
name: chaos-exporter-agent-1
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: chaos-exporter-agent-1
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: chaos-exporter-agent-1
name: chaos-exporter-agent-1
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: chaos-exporter-agent-1
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 1s
port: tcp
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
  • Chaos-exporter - chaos delegate -2
kind: Service
apiVersion: v1
metadata:
name: chaos-exporter-agent-2
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: chaos-exporter-agent-2
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: chaos-exporter-agent-2
name: chaos-exporter-agent-2
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: chaos-exporter-agent-2
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 1s
port: tcp
metricRelabelings:
- targetLabel: instance
replacement: 'chaos-exporter-service'
  • Kube state metrics exporter - chaos delegate -1
kind: Service
apiVersion: v1
metadata:
name: kube-state-metrics-agent-1
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: kube-state-metrics-agent-1
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kube-state-metrics-agent-1
name: kube-state-metrics-agent-1
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: kube-state-metrics-agent-1
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 30s
port: tcp
  • Kube state metrics exporter - chaos delegate -2
kind: Service
apiVersion: v1
metadata:
name: kube-state-metrics-agent-2
namespace: monitoring
spec:
type: ClusterIP
ports:
- name: tcp
port: 8080
targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
name: kube-state-metrics-agent-2
namespace: monitoring
subsets:
- addresses:
- ip: <AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
ports:
- name: tcp
port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kube-state-metrics-agent-2
name: kube-state-metrics-agent-2
namespace: monitoring
spec:
jobLabel: app
selector:
matchLabels:
app: kube-state-metrics-agent-2
namespaceSelector:
matchNames:
- monitoring
endpoints:
- interval: 30s
port: tcp

Detailed setup guide for Prometheus operator with service monitors

Resources

Learn more