Objective: The purpose of this tutorial is to walk you through the process of enabling the Elasticsearch interpreter for Zeppelin on the HDP 2.5 TP sandbox. As part of this process, we will install Elasticsearch and use Zeppelin to index and query data using Zeppelin and Elasticsearch. This is the first of two articles covering Elasticsearch on HDP. The second article covers pushing Twitter data to Elasticsearch using NiFi and provides a sample Zeppelin dashboard. You can find that article here: HCC Article Note: The Zeppelin Elasticserch interpreter is a community provided interpreter. It is not yet considered GA by Hortonworks and should only be used for development and testing purposes. Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). Note: While not required, I recommend using Vagrant to manage multiple versions of the Sandbox. Follow my tutorial here to set that up: HCC Article Scope: This tutorial was tested using the following environment and components: Mac OS X 10.11.6 HDP 2.5 Tech Preview on Hortonworks Sandbox Elasticsearch 2.3.5 and Elasticsearch 2.4.0 Note: This has also been tested on HDP 2.5 deployed with Cloudbreak on AWS. The specific steps may vary depending on your environment, but the high level process is the same. Steps: Here is the online documentation for the Elasticsearch interpreter for Zeppelin: Elasticseach Interpreter. If you follow the steps provided in this documentation, you will find that adding the Elasticserch interpreter is not possible as the documentation shows. That is because the interpreter is not enabled. If you try to add the interpreter, you will see it is not in the list. You should see something similar to: Verify Elasticsearch Interpreter is available The first thing we are going to do is ensure the Elasticsearch interpreter is available within the Zeppelin installation. You can verify the Elasticsearch intepreter is available by looking in the interpreter directory: $ ls -la /usr/hdp/current/zeppelin-server/interpreter/ total 76 drwxr-xr-x 19 zeppelin zeppelin 4096 2016-06-24 00:00 . drwxr-xr-x 8 zeppelin zeppelin 4096 2016-08-31 02:57 .. drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-23 23:59 alluxio drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-23 23:59 angular drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 cassandra drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 elasticsearch drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 file drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 flink drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 hbase drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 ignite drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 jdbc drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 kylin drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 lens drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 livy drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 md drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 psql drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 python drwxr-xr-x 2 zeppelin zeppelin 4096 2016-06-24 00:00 sh drwxr-xr-x 3 zeppelin zeppelin 4096 2016-06-24 00:00 spark Note: This process is easy on the sandbox. If you are using a different HDP environment, then you need to perform this step on the server on which Zeppelin is installed. If you do not see a directory for elasticsearch, you may have to run an interpreter install script. Here are the steps to run the interpreter install script: $ cd /usr/hdp/current/zeppelin-server/bin $ sudo ./install-interpreter.sh --name elasticsearch Add Elasticsearch Interpreter to the Zeppelin configuration Now we need to add the Elasticsearch interpreter to the Zeppelin configuration, which enables access to it. You need to modify the zeppelin.interpreters parameter. Click on the Zeppelin Notebook service in Ambari: Now, click on the Configs link: Expand Advanced zeppelin-config: Add the following string to the end of the zeppelin.interpreters parameter: ,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter Note: The comma is not a typo. It is required to seperate our added value from the previous value. It should look similar to this: Now click the Save button to save the settings. You should see an indication that you need to restart the Zeppelin service. It should look similar like this: Restart the Zeppelin Notebook service. Configure Zeppelin Interpreter Now you should be able to follow the documentation I linked previously for setting up the Elasticsearch interpreter. You should have something similar to this: The elasticsearch.host value will correspond to your ip address or sandbox.hortonworks.com if you have edited your local /etc/hosts file. Download Elasticsearch Now that Zeppelin is configured, we need to download Elasticsearch. The latest version is 2.4.0. You can read more about Elasticsearch here: Elasticsearch Website You can use curl to download Elasticsearch to your sandbox. $ cd ~ $ curl -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz Note: If you are using vagrant, you are able to download the file on your local computer and simply copy it to your Vagrant directory. The file will be visible within the sandbox in the /vagrant directory. Install Elasticsearch Next we need to extract Elasticsearch to /opt directory, which is where we'll run it. $ cd /opt $ sudo tar xvfz ~/elasticsearch-2.4.0.tar.gz Configure Elasticsearch We need to make a couple of changes to the Elasticsearch configuration file /opt/elasticsearch-2.4.0/config/elastiserach.yml. $ cd elasticsearch-2.4.0/config $ vi elasticsearch.yml We need to set the cluster.name setting to "elasticsearch". This is the default Zeppelin expects, however you can change this value in the Zeppelin configuration. cluster.name: elasticsearch We need to set the network.host setting to our sandbox hostname or ip. Elastic will default to binding to 127.0.0.1 which won't allow us to easily access it from outside of the sandbox. network.host: sandbox.hortonworks.com Make sure you have removed the # character at the start of the line for these two settings. Once you have completed these two changes, save the file: Press the esc key !wq Create Elasticsearch user We are going to create an elastic user to run the application. $ sudo useradd elastic -d /home/elastic Change Ownership of Elasticserach diretories We are going to change the ownership of the elastic directories to the elastic user: $ sudo chown -R elastic:elastsic /opt/elasticserach-2.4.0 Start elasticsearch We want to run Elasticsearch as the elastic user so first we'll switch to that user. $ sudo su - elastic $ cd /opt/elasticsearch-2.4.0 $ bin/elasticsearch You will see something similar to : $ bin/elasticsearch [2016-09-02 19:44:34,905][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed [2016-09-02 19:44:35,168][INFO ][node ] [Skyhawk] version[2.4.0], pid[22983], build[ce9f0c7/2016-08-29T09:14:17Z] [2016-09-02 19:44:35,168][INFO ][node ] [Skyhawk] initializing ... [2016-09-02 19:44:35,807][INFO ][plugins ] [Skyhawk] modules [lang-groovy, reindex, lang-expression], plugins [], sites [] [2016-09-02 19:44:35,856][INFO ][env ] [Skyhawk] using [1] data paths, mounts [[/ (/dev/mapper/vg_sandbox-lv_root)]], net usable_space [26.2gb], net total_space [42.6gb], spins? [possibly], types [ext4] [2016-09-02 19:44:35,856][INFO ][env ] [Skyhawk] heap size [990.7mb], compressed ordinary object pointers [true] [2016-09-02 19:44:35,856][WARN ][env ] [Skyhawk] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536] [2016-09-02 19:44:38,032][INFO ][node ] [Skyhawk] initialized [2016-09-02 19:44:38,032][INFO ][node ] [Skyhawk] starting ... [2016-09-02 19:44:38,115][INFO ][transport ] [Skyhawk] publish_address {172.28.128.4:9300}, bound_addresses {172.28.128.4:9300} [2016-09-02 19:44:38,119][INFO ][discovery ] [Skyhawk] elasticsearch/31d3OvlZT5WRnqYUW-GJwA [2016-09-02 19:44:41,157][INFO ][cluster.service ] [Skyhawk] new_master {Skyhawk}{31d3OvlZT5WRnqYUW-GJwA}{172.28.128.4}{172.28.128.4:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2016-09-02 19:44:41,206][INFO ][http ] [Skyhawk] publish_address {172.28.128.4:9200}, bound_addresses {172.28.128.4:9200} [2016-09-02 19:44:41,207][INFO ][node ] [Skyhawk] started [2016-09-02 19:44:41,223][INFO ][gateway ] [Skyhawk] recovered [0] indices into cluster_state Verify access to Elasticsearch Using your web browser, verify you get a response from Elasticsearch by using the following address: http://sandbox.hortonworks.com:9200 You should see something similar to: Alternatively, you can use curl: curl -XGET http://sandbox.hortonworks.com:9200 You will see a similar json output message. Add data to elasticsearch Now we are going to create a notebook in Zeppelin. You should have a note for each index operation in the notebook. Let's use the %elasticsearch and the index command to index some data: %elasticsearch index movies/default/1 { "title": "The Godfather", "director": "Francis Ford Coppola", "year": 1972, "genres": ["Crime", "Drama"] } %elasticsearch index movies/default/2 { "title": "Lawrence of Arabia", "director": "David Lean", "year": 1962, "genres": ["Adventure", "Biography", "Drama"] } %elasticsearch index movies/default/3 { "title": "To Kill a Mockingbird", "director": "Robert Mulligan", "year": 1962, "genres": ["Crime", "Drama", "Mystery"] } %elasticsearch index movies/default/4 { "title": "Apocalypse Now", "director": "Francis Ford Coppola", "year": 1979, "genres": ["Drama", "War"] } %elasticsearch index movies/default/5 { "title": "Kill Bill: Vol. 1", "director": "Quentin Tarantino", "year": 2003, "genres": ["Action", "Crime", "Thriller"] } %elasticsearch index movies/default/6 { "title": "The Assassination of Jesse James by the Coward Robert Ford", "director": "Andrew Dominik", "year": 2007, "genres": ["Biography", "Crime", "Drama"] } You should have a notebook that looks similar to this: For each of the index notes, click the play button to insert the data. Query Elasticsearch data Once the data is in Elasticseach, we can search using Zeppelin like this: %elasticsearch search /movies/default For this note, click the play button to run the query. You should see something similar to this: The Elasticsearch interpreter has great support for the Elasticsearch Query DSL (Domain Specific Language). You have the ability to easily filter the fields returned, create buckets and aggregations. Review: We have enabled the Elasticsearch interpreter in Zeppelin, indexed data into Elasticsearch and queried data from Elasticsearch using Zeppelin. Try indexing and querying data using your own data and using a different index name. ... View more