Elastic Logo

Elasticsearch Watcher Integration Guide

Watcher is the alerting and notification plugin for Elastisearch. It allows alerts to be created based on defined queries on Elasticsearch. TaskCall provides an endpoint where the alerts can be sent to trigger incidents allowing the underlying issue to be resolved faster.

Pointers
  • The integration happens between a TaskCall service and Elasticsearch Watcher.
  • Incidents will automatically resolve in TaskCall when issues are resolved in Elasticsearch.
  • At the moment this integration only supports alerts to be received from Elasticsearch Watcher to TaskCall.
In TaskCall
  1. Go to Configurations > Services . Select the service you want to integrate with.
  2. Once you are on the Service details page, go to the Integrations tab. Click on New Integration.
  3. Give the integration a name.
  4. From the integration types, select the top radio button indicating that you are trying to use a built-in integration.
  5. From the list of built-in integrations, select Prometheus.
  6. Click Save.
  7. Copy the Integration Key that is issued for the integration.
On Watcher
  1. Go to your Elasticsearch server.
  2. Make sure that you have Watcher set up and running on the server. You can verify by running the following curl command:

    curl -XGET 'http://localhost:9200/_watcher/stats?pretty'


    This should produce an output stating that Watcher has been started.

    {
       "_nodes" : {
         "total" : 1,
         "successful" : 1,
         "failed" : 0
       },
       "cluster_name" : "elasticsearch",
       "manually_stopped" : false,
       "stats" : [
         {
           "node_id" : "GOf0zK6FSEG-kxpAK5wh0g",
           "watcher_state" : "started",
           "watch_count" : 0,
           "execution_thread_pool" : {
             "queue_size" : 0,
             "max_size" : 0
           }
         }
      ]
    }


  3. Set up a watch with a Webhook for TaskCall and use a PUT request to add it to Watcher. Replace the <INTEGRATION_KEY> market in the body of attribute with the the Integration Key you copied over from TaskCall.

    curl -XPUT 'http://localhost:9200/_watcher/watch/cluster_health_watch' -H 'Content-Type: application/json' -d '{
       "trigger" : {
         "schedule" : { "interval" : "30m" }
       },
       "input" : {
         "http" : {
           "request" : {
             "host" : "localhost",
             "port" : 9200,
             "path" : "/_cluster/health"
           }
         }
       },
       "condition" : {
         "always" : {}
       },
       "actions" : {
         "taskcall": {
           "webhook": {
             "scheme": "https",
             "method": "POST",
             "host": "integrations.taskcallapp.com",
             "port": 443,
             "path": "/elasticsearch-watcher/<INTEGRATION_KEY>",
             "body": "{\"cluster\": \"{{ctx.payload.cluster_name}}\",\"watch_id\": \"{{ctx.watch_id}}\",\"status\": \"{{ctx.payload.status}}\",\"description\": \"Issue in elastic watcher\",\"message\": \"{{ctx.watch_id}}:{{ctx.payload.hits.total}}\"}",
             "headers": {"Content-type": "application/json"}
           }
         }
       }
    }'


  4. Please schedule the watch time, interval and conditions as per your need. The above is a minimal example of a configuration.
  5. The body attribute can be customized to include more information. However, it must always at least contain the following fields -- cluster, watch_id, status, description and message. TaskCall uses the description attribute as the incident title . while all the attributes of the body are displayed on the incident
  6. TaskCall only triggers incidents when the status of the alert is red. It automatically resolves the incident when the alert status changes to green.
On this page