Creating Custom ElasticSearch REST Action

Send to Kindle

code

The Mastering ElasticSearch book that is going to be published in December 2013 (probable date, but may be available earlier) will include a chapter that is dedicated to extending ElasticSearch. However, one of the topics – the simplest one didn’t made it into the book. Because of that we wanted to share this section of the book on the blog. We hope that you’ll find it useful.

Let’s start the journey of extending ElasticSearch with creating a custom REST action. We’ve chosen this as the first extension, because we wanted to take the simplest approach as the introduction to extending ElasticSearch. We assume that you already have a Java project created and that you are using Maven (its included in the Mastering ElasticSearch book).

The assumptions

In order to illustrate how to develop a custom REST action, we need to have some idea of how it should work. Our REST action will be really simple – it should return names of all the nodes or names of the nodes that start with the given prefix, if the prefix parameter is passed to it. In addition to that, it should only be available when using HTTP GET method, so POST requests for example shouldn’t be allowed.

Implementation details

We will need to develop two Java classes:

  • A class that extends BaseRestHandler ElasticSearch abstract class from org.elasticsearch.rest package that will be responded for handling the REST action code – we will call it a CustomRestAction.
  • A class that will be used by ElasticSearch to load the plugin – this class need to extend ElasticSearch AbstractPlugin class from org.elasticsearch.plugin package – we will call it CustomRestActionPlugin.

In addition to the two above we will need a simple text file that we will discuss after implementing the two above mentioned Java classes.

The REST action class

The most interesting class is the one that will be used to handle users requests – we will call it CustomRestAction. In order to work it needs to extend the BaseRestHandler class from the org.elasticsearch.rest package – the base class for REST actions in ElasticSearch. In order to extend that class, we need to implement the handleRequest method in which we will process user request and a three argument constructor that will be used to initialize the base class and register appropriate handler under which our REST action will be visible.
The whole code for CustomRestAction class looks as follows:

public class CustomRestAction extends BaseRestHandler {
 @Inject
 public CustomRestAction(Settings settings, Client client, RestController controller) {
  super(settings, client);
  controller.registerHandler(Method.GET, "/_mastering/nodes", this);
 }

 @Override
 public void handleRequest(RestRequest request, RestChannel channel) {
  String prefix = request.param("prefix", "");
  NodesInfoResponse response = client.admin().cluster().prepareNodesInfo().all().execute().actionGet();
  List<String> nodes = new ArrayList<String>();
  for (NodeInfo nodeInfo : response.getNodes()) {
   String nodeName = nodeInfo.getNode().getName();
   if (prefix.isEmpty()) {
    nodes.add(nodeName);
   } else if (nodeName.startsWith(prefix)) {
    nodes.add(nodeName);
   }
  }
  try {
   sendResponse(request, channel, nodes);
  } catch (IOException ioe) {
   logger.error("Error sending response", ioe);
  }
  return;
 }

 private void sendResponse(RestRequest request, RestChannel channel, List nodes) throws IOException {
  XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);
  builder.startObject().startArray("nodes");
  if (!nodes.isEmpty()) {
   builder.value(nodes);
  }
  builder.endArray().endObject();
  channel.sendResponse(new XContentRestResponse(request, RestStatus.OK, builder));
 }
}

The constructor

For each custom REST class ElasticSearch will pass three arguments when creating an object of such type – the Settings type object, which holds the settings, the Client type object which is an ElasticSearch client and the RestController type object we will use to bind our REST action to REST endpoint. The first two are required by the super class, so we invoke base class constructor and pass them.
There is one more thing – the @Inject annotation. It allows us to inform ElasticSearch that it should put the objects in the constructor during object creation. For more information about it, please refer to Javadoc of the mentioned annotation available at https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/inject/Inject.java.
Now let’s focus on the following code line:

controller.registerHandler(Method.GET, "/_mastering/nodes", this);

What it does it registers our custom REST action implementation and binds it to the endpoint of our choice. The first argument if the HTTP method type the REST action will be able to work with.   As we said earlier, we only want to respond to GET requests. If we would like to respond to multiple types of HTTP methods we should just include multiple registerHandler method invocations with each HTTP method. The second argument specifies the actual REST endpoint our custom action will be available at – in our case it will available under the /_mastering/nodes endpoint. The third argument is telling ElasticSearch which class should be responsible for handling the defined endpoint – in our case this is the class we are developing thus we are passing this.

Request handling

Although the handleRequest method is the longest one in our code it is not complicated. We start by reading the request parameter with the following line of code:

String prefix = request.param("prefix", "");

We store the prefix request parameter in the variable called prefix. By default we want an empty String object to be assigned to the prefix variable if there is no prefix parameter passed to the request (the default value is defined by the second parameter of param method of the request object).
Next, we retrieve the NodesInfoResponse object, by using ElasticSearch client object and its abilities to run administrative commands. The NodesInfoResponse object will contain an array of NodeInfo objects which we will use to get node names. What we need to do is return all the node names that starts with a given prefix or all if the prefix parameter was not present in the request. In order to do that we create a new array:

List<String> nodes = new ArrayList<String>();

And we iterate over the available nodes using the following for loop:

for (NodeInfo nodeInfo : response.getNodes())

We get the node name, by using the getName method of the DiscoveryNode object which is returned after invoking the getNode method of NodeInfo:

String nodeName = nodeInfo.getNode().getName();

If the prefix is empty or if it starts with the given prefix, we add the name of the node to the array we’ve created. After we iterated through all the NodeInfo objects we call the sendResponse method we’ve created. If that method throws an IOException type exception we write the reason of the exception to ElasticSearch logs.
Please note that in your production code you will probably want not only to store the error in the logs, but also return some kind of error to the request sender in order to inform him what is happening.

Response writing

The last thing regarding our CustomRestAction class is the response handling, which is the responsibility of the sendResponse method, which we created.  We want to send response in a proper JSON format just like ElasticSearch does. In order to do that we need a new instance of XContentBuilder object (from org.elasticsearch.common.xcontent package). To get it we use the following line of code:

XContentBuilder builder = RestXContentBuilder.restContentBuilder(request);

We pass the RestRequest type object to the static restContentBuilder method of the RestXContentBuilder class in order to get fully initialized XContentBuilder object we can use to construct our response in JSON. We use the XContentBuilder object we got to start JSON object (by using the startObject method) and to start an nodes array (by using the startArray method) inside that object that we will use to return matching nodes names. If the nodes array we’ve created is not empty we add those names as values to the array (using value method) and finally we end the array first (by calling endArray method) and then we close the object (by using endObject method).
After we have our XContentBuilder object ready to be sent as a response we use the sendResponse method of the RestChannel to send the response. In order to do that, we need to construct a new XContentRestResponse object. We do that in the following line:

channel.sendResponse(new XContentRestResponse(request, RestStatus.OK, builder));

As you can see, to create the XContentRestResponse object we need to pass three parameters – the RestRequest, the RestStatus and the XContentBuilder that holds our response. The RestStatus class allows us to specify the response code, which in our case is RestStatus.OK, because everything went smooth.

The plugin class

The CustomRestActionPlugin class will be the code that is used by ElasticSearch to initialize the plug-in itself. It extends the AbstractPlugin class from org.elasticsearch.plugin package. Because we are making an extension we are obliged to implement the following code parts:

  • constructor  – standard constructor that will take a single argument, in our case it will be empty,
  • onModule method – method that includes the code that will add our custom REST action so that ElasticSearch will know about it,
  • name method – name of our plugin,
  • description method – short description of our plugin.

The whole class code looks like this:

public class CustomRestActionPlugin extends AbstractPlugin {
 @Inject
 public CustomRestActionPlugin(Settings settings) {
 }

 public void onModule(RestModule module) {
  module.addRestAction(CustomRestAction.class);
 }

 @Override
 public String name() {
  return "CustomRestActionPlugin";
 }

 @Override
 public String description() {
  return "Custom REST action";
 }
}

The constructor, name and description methods are very simple and we will just skip discussing them and we will focus on the onModule method. This method takes a single argument – the RestModule class object, which is the class that allows us to register our custom REST action. ElasticSearch will call the onModule method for all the modules that are available and eligible (all REST actions). What we do is just a simple call to RestModule addRestAction method passing in our CustomRestAction class as an argument. And that’s all when it comes to Java development.

Informing ElasticSearch about our REST action

We have our code ready, but we need one additional thing – we need to let ElasticSearch know what the class registering our plugin is – the one we’ve called CustomRestActionPlugin. In order to do that, we create an es-plugin.properties file in the src/main/resources directory with the following content:

plugin=pl.solr.rest.CustomRestActionPlugin

We just specify the plugin property there, which should have a value of the class we use for register our plugins (the one that extends the ElasticSearch AbstractPlugin class). This file will be included in the jar file that will be created during the build process and will be used by ElasticSearch during plugin load process.

It is time for testing

Of course we could leave it now and say that we are done, but we won’t. We would like to show you how to build each of the plug-ins, install it and finally test it to see if it actually works. Let’s start with building our plugin.

Building the REST action plugin

We start with the easiest part – building our plug-in. In order to do that, we run a simple command:

mvn compile package

We tell Maven that we want the code to be compiled and packaged. After the command finishes we can find the archive with the plugin in the target/release directory (assuming you are using similar project setup to the one we’ve describe in the beginning of the chapter).

Installing the REST action plugin

In order to install the plug-in we will use the plugin command that is located in the bin directory of ElasticSearch distributable package. Assuming that we have our plug-in archive stored in /home/install/es/plugins directory we would run the following command (we run it from ElasticSearch home directory):

bin/plugin --install rest --url file:/home/install/es/plugins/elasticsearch-rest-0.90.3.zip

We need to install the plugin on all the nodes in our cluster, because we want to be able to run our custom REST action on each ElasticSearch instance.
In order to learn more about installing ElasticSearch plugins please refer to our previous book – ElasticSearch Server or to the official ElasticSearch documentation http://www.elasticsearch.org/guide/reference/modules/plugins/.

After we’ve have the plug-in installed we need to restart our ElasticSearch instance we were making the installation on. After restart we should see something like this in the logs:

[2013-08-21 21:04:48,348][INFO ][plugins] [Archer] loaded [CustomRestActionPlugin], sites []

As you can see, ElasticSearch informed us that the plugin named CustomRestActionPlugin was loaded.

Checking if the REST action plugin works

We can finally check if the plugin works. In order to do that we will run the following command:

curl -XGET 'localhost:9200/_mastering/nodes?pretty'

In result we should get all the nodes in the cluster, because we didn’t provide the prefix parameter and this is exactly what we’ve got from ElasticSearch:

{
"nodes" : [ "Archer" ]
}

Because we only had one node in our ElasticSearch cluster, we’ve got the nodes array with only a single entry.
Now let’s test what will happen if we will add the prefix=Are parameter to our request. The exact command we’ve used was as follows:

curl -XGET 'localhost:9200/_mastering/nodes?prefix=Are&pretty'

The response from ElasticSearch was as follows:

{
"nodes" : [ ]
}

As you can see, the nodes array is empty, because we don’t have any node in the cluster, that would start with the Are prefix.

One thought on “Creating Custom ElasticSearch REST Action

  1. Luiz Henrique Zambom Santana says:

    Hello:

    I am trying to develop a simple plugin to count terms in a repository using Lucene and expose the result through a elastic search RestHandler.

    I successfully went through the tutorial until the parte of starting the plugin.

    And I can see that my plugin is installed and loaded.

    [2013-11-06 14:47:00,471][INFO ][org.elasticsearch.plugin.wordfrequencyplugin.WordFrequencyPlugin] WordFrequencyPlugin settings
    [2013-11-06 14:47:00,472][INFO ][org.elasticsearch.plugin.wordfrequencyplugin.WordFrequencyPlugin] name
    [2013-11-06 14:47:00,475][INFO ][plugins ] [Starbolt] loaded [WordFrequencyPlugin], sites []
    [2013-11-06 14:47:00,554][INFO ][org.elasticsearch.plugin.wordfrequencyplugin.WordFrequencyPlugin] onModule
    [2013-11-06 14:47:02,200][INFO ][org.elasticsearch.plugin.wordfrequencyplugin.CustomRestRegisterAction] Registering action /_counter/movies

    However, when I try:

    lsantana@abc:~/elasticsearch-0.90.6/bin$ curl -XGET ‘localhost:9200/_counter/movies?term=XPTO’
    No handler found for uri [/_counter/movies?term=XPTO] and method [GET]lsantana@iqserdev10:~/elasticsearch-0.90.6/bin$

    My BaseRestHandler looks like this:

    public class CustomRestRegisterAction extends BaseRestHandler {

    private static Logger log = Logger.getLogger(CustomRestRegisterAction.class);

    public CustomRestRegisterAction(Settings settings, Client client) {
    super(settings, client);
    }

    @Inject
    public CustomRestRegisterAction(Settings settings, Client client,
    RestController controller) {

    super(settings, client);

    String path = “/_counter/movies”;

    controller.registerHandler(Method.GET, path, new CustomRestRegisterAction(settings, client));
    log.info(“Registering action “+path);
    }

    public void handleRequest(RestRequest request, RestChannel channel) {

    log.info(“Handling request”);

    String term = request.param(“term”);

    Lucene l = new Lucene();
    int frequency = -1;
    try {

    DirectoryReader directoryReader = new DirectoryReader();
    Directory directory = directoryReader.getDocuments();

    frequency = l.frequency(term, “city”, directory);
    } catch (IOException e) {
    e.printStackTrace();
    }

    channel.sendResponse(new StringRestResponse(RestStatus.OK, String
    .valueOf(frequency)));
    }
    }

    Any clue about what can be missing?

    Best regards,

    Luiz

Leave a Reply