Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Super Guru

Objective:

The purpose of this tutorial is to walk you through the process of enabling the Elasticsearch interpreter for Zeppelin on the HDP 2.5 TP sandbox. As part of this process, we will install Elasticsearch and use Zeppelin to index and query data using Zeppelin and Elasticsearch.

This is the first of two articles covering Elasticsearch on HDP. The second article covers pushing Twitter data to Elasticsearch using NiFi and provides a sample Zeppelin dashboard. You can find that article here: HCC Article

Note: The Zeppelin Elasticserch interpreter is a community provided interpreter. It is not yet considered GA by Hortonworks and should only be used for development and testing purposes.

Prerequisites:

You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview).

Note: While not required, I recommend using Vagrant to manage multiple versions of the Sandbox. Follow my tutorial here to set that up: HCC Article

Scope:

This tutorial was tested using the following environment and components:

  • Mac OS X 10.11.6
  • HDP 2.5 Tech Preview on Hortonworks Sandbox
  • Elasticsearch 2.3.5 and Elasticsearch 2.4.0

Note: This has also been tested on HDP 2.5 deployed with Cloudbreak on AWS. The specific steps may vary depending on your environment, but the high level process is the same.

Steps:

Here is the online documentation for the Elasticsearch interpreter for Zeppelin: Elasticseach Interpreter. If you follow the steps provided in this documentation, you will find that adding the Elasticserch interpreter is not possible as the documentation shows. That is because the interpreter is not enabled.

If you try to add the interpreter, you will see it is not in the list. You should see something similar to:

7328-zeppeling-interpreter-list.png

Verify Elasticsearch Interpreter is available

The first thing we are going to do is ensure the Elasticsearch interpreter is available within the Zeppelin installation. You can verify the Elasticsearch intepreter is available by looking in the interpreter directory:

$ ls -la /usr/hdp/current/zeppelin-server/interpreter/
total 76
drwxr-xr-x 19 zeppelin zeppelin 4096 2016-06-24 00:00 .
drwxr-xr-x  8 zeppelin zeppelin 4096 2016-08-31 02:57 ..
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-23 23:59 alluxio
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-23 23:59 angular
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 cassandra
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 elasticsearch
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 file
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 flink
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 hbase
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 ignite
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 jdbc
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 kylin
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 lens
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 livy
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 md
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 psql
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 python
drwxr-xr-x  2 zeppelin zeppelin 4096 2016-06-24 00:00 sh
drwxr-xr-x  3 zeppelin zeppelin 4096 2016-06-24 00:00 spark

Note: This process is easy on the sandbox. If you are using a different HDP environment, then you need to perform this step on the server on which Zeppelin is installed.

If you do not see a directory for elasticsearch, you may have to run an interpreter install script. Here are the steps to run the interpreter install script:

$ cd /usr/hdp/current/zeppelin-server/bin
$ sudo ./install-interpreter.sh --name elasticsearch

Add Elasticsearch Interpreter to the Zeppelin configuration

Now we need to add the Elasticsearch interpreter to the Zeppelin configuration, which enables access to it. You need to modify the zeppelin.interpreters parameter.

Click on the Zeppelin Notebook service in Ambari:

7303-ambari-1.png

Now, click on the Configs link:

7304-ambari-2.png

Expand Advanced zeppelin-config:

7305-ambari-3.png

Add the following string to the end of the zeppelin.interpreters parameter:

,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter

Note: The comma is not a typo. It is required to seperate our added value from the previous value.

It should look similar to this:

7306-ambari-4.png

Now click the Save button to save the settings. You should see an indication that you need to restart the Zeppelin service. It should look similar like this:

7307-ambari-5.png

Restart the Zeppelin Notebook service.

Configure Zeppelin Interpreter

Now you should be able to follow the documentation I linked previously for setting up the Elasticsearch interpreter. You should have something similar to this:

7310-zeppelin-configuration.png

The elasticsearch.host value will correspond to your ip address or sandbox.hortonworks.com if you have edited your local /etc/hosts file.

Download Elasticsearch

Now that Zeppelin is configured, we need to download Elasticsearch. The latest version is 2.4.0. You can read more about Elasticsearch here: Elasticsearch Website

You can use curl to download Elasticsearch to your sandbox.

$ cd ~
$ curl -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2...

Note: If you are using vagrant, you are able to download the file on your local computer and simply copy it to your Vagrant directory. The file will be visible within the sandbox in the /vagrant directory.

Install Elasticsearch

Next we need to extract Elasticsearch to /opt directory, which is where we'll run it.

$ cd /opt
$ sudo tar xvfz ~/elasticsearch-2.4.0.tar.gz

Configure Elasticsearch

We need to make a couple of changes to the Elasticsearch configuration file /opt/elasticsearch-2.4.0/config/elastiserach.yml.

$ cd elasticsearch-2.4.0/config
$ vi elasticsearch.yml

We need to set the cluster.name setting to "elasticsearch". This is the default Zeppelin expects, however you can change this value in the Zeppelin configuration.

cluster.name: elasticsearch

We need to set the network.host setting to our sandbox hostname or ip. Elastic will default to binding to 127.0.0.1 which won't allow us to easily access it from outside of the sandbox.

network.host: sandbox.hortonworks.com

Make sure you have removed the # character at the start of the line for these two settings. Once you have completed these two changes, save the file:

Press the esc key
!wq

Create Elasticsearch user

We are going to create an elastic user to run the application.

$ sudo useradd elastic -d /home/elastic

Change Ownership of Elasticserach diretories

We are going to change the ownership of the elastic directories to the elastic user:

$ sudo chown -R elastic:elastsic /opt/elasticserach-2.4.0

Start elasticsearch

We want to run Elasticsearch as the elastic user so first we'll switch to that user.

$ sudo su - elastic
$ cd /opt/elasticsearch-2.4.0
$ bin/elasticsearch

You will see something similar to :

$ bin/elasticsearch
[2016-09-02 19:44:34,905][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-09-02 19:44:35,168][INFO ][node                     ] [Skyhawk] version[2.4.0], pid[22983], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-09-02 19:44:35,168][INFO ][node                     ] [Skyhawk] initializing ...
[2016-09-02 19:44:35,807][INFO ][plugins                  ] [Skyhawk] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-09-02 19:44:35,856][INFO ][env                      ] [Skyhawk] using [1] data paths, mounts [[/ (/dev/mapper/vg_sandbox-lv_root)]], net usable_space [26.2gb], net total_space [42.6gb], spins? [possibly], types [ext4]
[2016-09-02 19:44:35,856][INFO ][env                      ] [Skyhawk] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-09-02 19:44:35,856][WARN ][env                      ] [Skyhawk] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-09-02 19:44:38,032][INFO ][node                     ] [Skyhawk] initialized
[2016-09-02 19:44:38,032][INFO ][node                     ] [Skyhawk] starting ...
[2016-09-02 19:44:38,115][INFO ][transport                ] [Skyhawk] publish_address {172.28.128.4:9300}, bound_addresses {172.28.128.4:9300}
[2016-09-02 19:44:38,119][INFO ][discovery                ] [Skyhawk] elasticsearch/31d3OvlZT5WRnqYUW-GJwA
[2016-09-02 19:44:41,157][INFO ][cluster.service          ] [Skyhawk] new_master {Skyhawk}{31d3OvlZT5WRnqYUW-GJwA}{172.28.128.4}{172.28.128.4:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-09-02 19:44:41,206][INFO ][http                     ] [Skyhawk] publish_address {172.28.128.4:9200}, bound_addresses {172.28.128.4:9200}
[2016-09-02 19:44:41,207][INFO ][node                     ] [Skyhawk] started
[2016-09-02 19:44:41,223][INFO ][gateway                  ] [Skyhawk] recovered [0] indices into cluster_state

Verify access to Elasticsearch

Using your web browser, verify you get a response from Elasticsearch by using the following address:

http://sandbox.hortonworks.com:9200

You should see something similar to:

7308-browser.png

Alternatively, you can use curl:

curl -XGET http://sandbox.hortonworks.com:9200

You will see a similar json output message.

Add data to elasticsearch

Now we are going to create a notebook in Zeppelin. You should have a note for each index operation in the notebook. Let's use the %elasticsearch and the index command to index some data:

%elasticsearch
index movies/default/1 {
    "title": "The Godfather",
    "director": "Francis Ford Coppola",
    "year": 1972,
    "genres": ["Crime", "Drama"]
}
%elasticsearch
index movies/default/2 {
    "title": "Lawrence of Arabia",
    "director": "David Lean",
    "year": 1962,
    "genres": ["Adventure", "Biography", "Drama"]
}
%elasticsearch
index movies/default/3 {
    "title": "To Kill a Mockingbird",
    "director": "Robert Mulligan",
    "year": 1962,
    "genres": ["Crime", "Drama", "Mystery"]
}
%elasticsearch
index movies/default/4 {
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "War"]
}
%elasticsearch
index movies/default/5 {
    "title": "Kill Bill: Vol. 1",
    "director": "Quentin Tarantino",
    "year": 2003,
    "genres": ["Action", "Crime", "Thriller"]
}
%elasticsearch
index movies/default/6 {
    "title": "The Assassination of Jesse James by the Coward Robert Ford",
    "director": "Andrew Dominik",
    "year": 2007,
    "genres": ["Biography", "Crime", "Drama"]
}

You should have a notebook that looks similar to this:

7309-zeppelin-notebook-1.png

For each of the index notes, click the play button to insert the data.

Query Elasticsearch data

Once the data is in Elasticseach, we can search using Zeppelin like this:

%elasticsearch
search /movies/default

For this note, click the play button to run the query. You should see something similar to this:

7321-zeppelin-notebook-2.png

The Elasticsearch interpreter has great support for the Elasticsearch Query DSL (Domain Specific Language). You have the ability to easily filter the fields returned, create buckets and aggregations.

Review:

We have enabled the Elasticsearch interpreter in Zeppelin, indexed data into Elasticsearch and queried data from Elasticsearch using Zeppelin. Try indexing and querying data using your own data and using a different index name.

7,599 Views
Version history
Last update:
‎08-17-2019 10:27 AM
Updated by:
Contributors