Livy server on CDH 6.x.x
Installing and configuring Livy on CDH 6.x.x
Livy is a preferred way to run Spark jobs on several Hadoop installations, but not on CDH. While preparing for a CDP migration, one of our use-cases switched to Apache Airflow to run jobs without requiring an edge node or "bastion node" and they wanted to begin using Airflow before the CDP migration, so they asked me to install Livy on a CDH edge node.
A search online for Livy on CDH returned little helpful information, but I did find information on how to download and install it at https://livy.apache.org/
Step 1: Determine which account will be used to run Livy
Linux security will allow an application to access or execute any program or file the executing account can access unless you configure selinux or another access management software. Pick an account to run Livy from.
Step 2: Set up a keytab
You'll need a Kerberos principal. If you use Active Directory principals with your CDH deployment, then this account will be outside of your Hadoop platform. You can use a tool like ktutil to create a keytab for your Kerberos principal.
Step 3: Set up your server to run Livy
Livy requires the basic Hadoop and Spark environment variables.
export JAVA_HOME=/usr/java/default/jre
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
Step 4: Download and install Livy
- Download the Livy package zip file from https://livy.apache.org/download/ using wget.
Example:cd /var/tmp
wget https://dlcdn.apache.org/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip - Unzip the resulting zip file
unzip apache-livy-0.7.1-incubating-bin.zip - Deploy the package and link the default symlink directory
mkdir /opt/livy
mv /var/tmp/apache-livy-0.7.1-incubating-bin /opt/livy/
ln -s /opt/livy/apache-livy-0.7.1-incubating-bin /opt/livy/default
Step 5: Set up livy.conf
- Livy places template files that you'll need to copy to "real" files. You need to configure livy.conf.
cp /opt/livy/default/conf/livy.conf.template /opt/livy/default/conf/livy.conf - Edit the newly created livy.conf file and add two lines in the commented-out Kerberos section.
livy.server.launch.kerberos.principal=${KERBEROS_PRINCIPAL}
livy.server.launch.kerberos.keytab=${KERBEROS KEYTAB} - Specify the full Kerberos principal name and the full path to the keytab.
Step 6: Run the Livy server
Livy server runs as a background process. This article doesn't discuss how to run it as a service that starts automatically.
/opt/livy/default/bin/livy-server start
Step 7: Test Livy
You can use one of the recommended test commands from another node:
curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" http://<LIVY_HOST>:8998/sessions
You can also test from a web browser:
http://<LIVY_HOST>:8998
Disclaimer: This article is contributed by an external user. The steps may not be verified by Cloudera and may not be applicable for all use cases and may be very specific to a particular distribution. Please follow with caution and at your own risk. If needed, raise a support case to get the confirmation.
