Reroute API explained

Send to Kindle

routeTill now, we didn’t talk about shard allocation process or ability to move shards around the cluster. If you recall what shard allocation is – the process of allocating shards to nodes which can happen during initial cluster start, during replicas allocation or when nodes are being removed or added to the cluster. However apart from the allocation mechanism control, sometimes it is crucial to be able to cancel shard allocation or move shards around inside the cluster. In order to do that ElasticSearch exposes the reroute API, which allows us to move shards, cancel their allocation and finally force the allocation. You can imagine a situation when you would like to take off a single node from your cluster for maintenance, but you have data on it and first you want to somehow move the data to some other node. This is where the reroute API can come in handy. So now, let’s discuss all the possibilities that give us.

Example cluster

Let’s start with a simple cluster setup. In order to illustrate all the examples we’ll create a three nodes cluster. After that, we’ll run the following command to create a simple index:

curl -XPOST 'localhost:9200/mastering' -d '{
 "settings" : {
  "index" : {
   "number_of_shards" : 2,
   "number_of_replicas" : 1
  }
 }
}'

The above command will create an index called mastering with two shards and a replica for each shard. Now let’s check the status of the newly created index by running the following command:

curl -XGET 'localhost:9200/mastering/_status?pretty'

The response to the above command is quite large. What we are interested in though is where the shards and replicas were created. The primary shard numbered 0 was assigned to the node with identifier 6GVd-ktcS2um4uM4AAJQhQ. The response part responsible for that information is as follows:

"routing" : {
 "state" : "STARTED",
 "primary" : true,
 "node" : "6GVd-ktcS2um4uM4AAJQhQ",
 "relocating_node" : null,
 "shard" : 0,
 "index" : "mastering"
}

The primary shard numbered 1 was assigned to the node with identifier iw76Z_TaTfGRmbtCcPHF0Q. The response part responsible for that information is as follows:

"routing" : {
 "state" : "STARTED",
 "primary" : true,
 "node" : "iw76Z_TaTfGRmbtCcPHF0Q",
 "relocating_node" : null,
 "shard" : 1,
 "index" : "mastering"
}

The replica shard for the shard numbered 0 was assigned to the node with identifier wJq0kPSHTHCovjuCsVK0-A. The response part responsible for that information is as follows:

"routing" : {
 "state" : "STARTED",
 "primary" : false,
 "node" : "wJq0kPSHTHCovjuCsVK0-A",
 "relocating_node" : null,
 "shard" : 0,
 "index" : "mastering"
}

The replica shard for the shard numbered 1 was assigned to the node with identifier wJq0kPSHTHCovjuCsVK0-A. The response part responsible for that information is as follows:

"routing" : {
 "state" : "STARTED",
 "primary" : false,
 "node" : "wJq0kPSHTHCovjuCsVK0-A",
 "relocating_node" : null,
 "shard" : 1,
 "index" : "mastering"
}

Please note that the nodes identifiers shown in the examples will be different than the ones on your cluster, so please consider that when running the examples. So our cluster looks like this:
1435OS_4_1

Moving shards around the cluster

Let’s start with learning how to move shards around. For that we will try to  move a single replica shard from node with identifier wJq0kPSHTHCovjuCsVK0-A to a node with identifier iw76Z_TaTfGRmbtCcPHF0Q. In order to do that, we would send the following commands:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [ {
  "move" : {
   "index" : "mastering", "shard" : 0, "from_node" : "wJq0kPSHTHCovjuCsVK0-A", "to_node" : "iw76Z_TaTfGRmbtCcPHF0Q"
  }
 } ]
}'

Let’s discuss the command that we’ve just sent. First of all, we’ve sent it to the _cluster/reroute REST end-point, which is responsible for handling shard allocation commands. The reroute API accept an array of reroute commands, which are grouped together with the use of array named commands. The available commands are:

  • move – to move shard from one node to another,
  • allocate – to force shard allocation on a particular node,
  • cancel – to cancel shard allocation on a particular node.

For now let’s focus on the first command – the move one. As you can see we’ve included a move command for the shard with the number 0. The parameters available for the move command are as follows:

  • index – the name of the index we are going to manipulate shards for,
  • shard – shard number,
  • from_node – from which node the shard should be moved,
  • to_node – which node the shard should be moved to.

After running the above command and waiting for ElasticSearch to finish its work, we can again run a command to see that status of the index and we would see, that the command worked (we’ve stored the results of that command in a file called status_after_move.json). This is how the cluster can be visualized after the move command finished running:

1435OS_4_2

Forcing shard allocation

In addition to the move command, which enables us to move shards between nodes, ElasticSearch enables us to force allocation of a particular shard. We can use that to enforce index shard allocation that is living in the unallocated state (for example after an error occurred). In order to do that, similar to what we did before, we need to send a request to the  _cluster/reroute REST end-point. For example if we would like to force allocation of a replica shard, that has a number of 0, that belongs to the mastering index and we want to allocate it to a node with identifier wJq0kPSHTHCovjuCsVK0-A we would run the following request:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [ {
  "allocate" : {
   "index" : "mastering", "shard" : 0, "node" : "wJq0kPSHTHCovjuCsVK0-A"
  }
 } ]
}'

The parameters available for the allocate command are as follows:

  • index – the name of the index we are going to manipulate shards for,
  • shard – shard number,
  • node – on which node the allocation should be forced,
  • allow_primary – a Boolean value that defaults to false and tells ElasticSearch if the allocate command is allowed to be run on a primary shard. This is set to false by default in order not to result in cluster being inconsistent because of forcing primary shard allocation. Please be aware that setting this parameter to true may results in a data loss.

Cancelling shard allocation

In addition to the two above described command ElasticSearch enables us to cancel allocation of a particular shard. Please remember that when you cancel a shard that is being allocated it will remain in unallocated state, which can cause your cluster to be not operative. This can come in handy when your shard allocation is costly and you want to cancel the allocation that is currently in progress, for example to allocate different shard first. In order to do that, we need to send a request to the  _cluster/reroute REST end-point. For example if we would like to cancel ongoing allocation of a replica shard, that has a number of 1, that belongs to the mastering index and is being allocated to a node with identifier wJq0kPSHTHCovjuCsVK0-A we would run the following request:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [ {
  "cancel" : {
   "index" : "mastering", "shard" : 1, "node" : "wJq0kPSHTHCovjuCsVK0-A"
  }
 } ]
}'

The parameters available for the cancel command are as follows:

  • index – the name of the index we are going to manipulate shards for,
  • shard – shard number,
  • node – on which node the cancel operation should be performed,
  • allow_primary – a Boolean value that defaults to false and tells ElasticSearch if the cancel command is allowed to be run on a primary shard. This is set to false by default in order not to result in cluster being inconsistent because of canceling primary shard allocation.

Blog post background

The above post was a part of the incoming Mastering ElasticSearch book we are finishing for Packt Publishing. However, because we had to make it in the agreed page count number, we had to chose which part will be left aside. The Cluster Reroute API description was one of them. We will continue to post fragment that had to be removed and we hope you can make use of them :)

Tagged , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>