Datadog probe

Datadog probe allows you to query a Datadog Synthetic test and use its results through the fault's chaos duration to evaluate the probe outcome.

• Both API tests and Browser tests are supported.
• The probe may only be executed in the EOT mode, as the probe evaluation is based on the result of all the test iterations executed through the fault chaos duration.

note

If there are no iterations of the synthetics test through the chaos duration of the fault, the probe is marked as failed.

Providing secrets

Datadog secret keys need to be provided prior to using the probe, which are used to authenticate with the Datadog APIs. This includes an API key and an Application key.

Kubernetes

For a Kubernetes chaos infrastructure, the secrets shall be provided using a Kubernetes secret of the following format:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  type: Opaque
stringData:
  DD_API_KEY: "xxxxxxxxxxxxxxxxxxxx"
  DD_APP_KEY: "xxxxxxxxxxxxxxxxxxxx"

The secret name, that is, datadog-secret has to be provided while configuring the probe using the datadogCredentialsSecretName field.

Linux

For a Linux chaos infrastructure, the secrets are provided using an environment file at the following path, which is located on the machine where the infrastructure executes:

/etc/linux-chaos-infrastructure/datadog.env

DD_API_KEY="xxxxxxxxxxxxxxxxxxxx"
DD_APP_KEY="xxxxxxxxxxxxxxxxxxxx"

Probe definition

Kubernetes

For a Kubernetes chaos infrastructure, the probe is defined at .spec.experiments[].spec.probe path in the chaos engine manifest:

kind: Workflow
apiVersion: argoproj.io/v1alpha1
spec:
  templates:
    - inputs:
        artifacts:
          - raw:
              data: |
                apiVersion: litmuschaos.io/v1alpha1
                kind: ChaosEngine
                spec:
                  experiments:
                    - spec:
                        probe:
                          ####################################
                          Probes are defined here
                          ####################################

Linux

For a Linux chaos infrastructure, the probe is defined at .spec.tasks[].definition.chaos.probes path in the Linux chaos experiment manifest:

apiVersion: litmuschaos.io/v1alpha1
kind: LinuxChaosExperiment
spec:
  tasks:
    - name: task-1
      definition:
        chaos:
          probes:
            ####################################
            Probes are defined here
            ####################################

Schema

Listed below is the Datadog Probe schema with common properties shared across all probes and properties unique to Datadog probe.

Field	Description	Type	Range	Notes
name	Flag to hold the name of the probe	Mandatory	N/A type: string	The name holds the name of the probe. It can be set based on the use-case
type	Flag to hold the type of the probe	Mandatory	httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe	The type supports four types of probes. It can one of the httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe
mode	Flag to hold the mode of the probe	Mandatory	SOT, EOT, Edge, Continuous, OnChaos	The mode supports five modes of probes. It can one of the SOT, EOT, Edge, Continuous, and OnChaos. For Datadog probe only EOT mode is supported
datadogSite	Site for datadog probe	Mandatory	datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, datadoghq.eu, ddog-gov.com, and ap1.datadoghq.com	The datadogSite supports six values. Refer here for details
datadogCredentialsSecretName	Name of the secret having datadog probe secret keys	Optional	N/A type: string	Name of the Kubernetes secret containing the Datadog secret keys. Only required for Kubernetes chaos infrastructure

Synthetics test

Field	Description	Type	Range	Notes
publicId	Public ID of the synthetic test	Mandatory	N/A type: string	The publicId holds the ID of the synthetic test.
testType	Type of the synthetic test	Mandatory	api, browser	The testType holds the type of the synthetic test. It can one of api and browser

Run properties

Field	Description	Type	Range	Notes
probeTimeout	Flag to hold the timeout of the probe	Mandatory	N/A type: string	The probeTimeout represents the time limit for the probe to execute the specified check and return the expected data
attempt	Flag to hold the attempt of the probe	Mandatory	N/A type: integer	The attempt contains the number of times a check is run upon failure in the previous attempts before declaring the probe status as failed.
interval	Flag to hold the interval of the probe	Mandatory	N/A type: string	The interval contains the interval for which probes waits between subsequent retries
probePollingInterval	Flag to hold the polling interval for the probes (applicable for all modes)	Optional	N/A type: string	The probePollingInterval contains the time interval for which continuous and onchaos probe should be sleep after each iteration
initialDelaySeconds	Flag to hold the initial delay interval for the probes	Optional	N/A type: integer	The initialDelaySeconds represents the initial waiting time interval for the probes.
stopOnFailure	Flags to hold the stop or continue the experiment on probe failure	Optional	N/A type: boolean	The stopOnFailure can be set to true/false to stop or continue the experiment execution after probe fails

Definition

Kubernetes

probe:
  - name: datadog-probe
    type: "DatadogProbe"
    mode: "EOT"
    datadogProbe/inputs:
      datadogSite: us5.datadoghq.com
      syntheticsTest:
        publicId: zgs-mq8-pgy
        testType: api
      datadogCredentialsSecretName: dd-secret
    runProperties:
      probeTimeout: 2s
      attempt: 1
      interval: 3s
      stopOnFailure: false

Linux

probes:
  - name: datadog-probe
    type: "DatadogProbe"
    mode: "EOT"
    datadogProbe/inputs:
      datadogSite: us5.datadoghq.com
      syntheticsTest:
        publicId: zgs-mq8-pgy
        testType: api
    runProperties:
      probeTimeout: 2s
      attempt: 1
      interval: 3s
      stopOnFailure: false

API test

To trigger an API test, specify the syntheticsTest.testType as api.

Use the following example to tune this:

Kubernetes

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          syntheticsTest:
            publicId: zgs-mq8-pgy
            testType: api
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false

Linux

apiVersion: litmuschaos.io/v1alpha1
kind: LinuxChaosExperiment
metadata:
  name: process-kill
  labels:
    context: process-kill
    name: process-kill
spec:
  steps:
    - - name: process-kill-task
  tasks:
    - name: process-kill-task
      taskType: "chaos"
      weight: 10
      definition:
        chaos:
          probes:
            - name: datadog-probe
              type: "DatadogProbe"
              mode: "EOT"
              datadogProbe/inputs:
                datadogSite: us5.datadoghq.com
                syntheticsTest:
                  publicId: zgs-mq8-pgy
                  testType: api
              runProperties:
                probeTimeout: 2s
                attempt: 1
                interval: 3s
                stopOnFailure: false
          experiment: linux-process-kill
          processKillChaos/inputs:
            duration: 30
            processNames: "nginx"
            forceKill: false

Browser test

To trigger a browser test, specify the syntheticsTest.testType as browser.

Use the following example to tune this:

Kubernetes

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          syntheticsTest:
            publicId: zgs-mq8-pgy
            testType: browser
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false

Linux

apiVersion: litmuschaos.io/v1alpha1
kind: LinuxChaosExperiment
metadata:
  name: process-kill
  labels:
    context: process-kill
    name: process-kill
spec:
  steps:
    - - name: process-kill-task
  tasks:
    - name: process-kill-task
      taskType: "chaos"
      weight: 10
      definition:
        chaos:
          probes:
            - name: datadog-probe
              type: "DatadogProbe"
              mode: "EOT"
              datadogProbe/inputs:
                datadogSite: us5.datadoghq.com
                syntheticsTest:
                  publicId: zgs-mq8-pgy
                  testType: browser
              runProperties:
                probeTimeout: 2s
                attempt: 1
                interval: 3s
                stopOnFailure: false
          experiment: linux-process-kill
          processKillChaos/inputs:
            duration: 30
            processNames: "nginx"
            forceKill: false