Installing and configuring Livy on CDH 6.x.x Livy is a preferred way to run Spark jobs on several Hadoop installations, but not on CDH. While preparing for a CDP migration, one of our use-cases switched to Apache Airflow to run jobs without requiring an edge node or "bastion node" and they wanted to begin using Airflow before the CDP migration, so they asked me to install Livy on a CDH edge node.   A search online for Livy on CDH returned little helpful information, but I did find information on how to download and install it at https://livy.apache.org/ Step 1: Determine which account will be used to run Livy Linux security will allow an application to access or execute any program or file the executing account can access unless you configure selinux or another access management software. Pick an account to run Livy from. Step 2: Set up a keytab You'll need a Kerberos principal. If you use Active Directory principals with your CDH deployment, then this account will be outside of your Hadoop platform. You can use a tool like ktutil to create a keytab for your Kerberos principal.   Step 3: Set up your server to run Livy Livy requires the basic Hadoop and Spark environment variables.   export JAVA_HOME=/usr/java/default/jre export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark   Step 4: Download and install Livy Download the Livy package zip file from https://livy.apache.org/download/ using wget. Example: cd /var/tmp wget https://dlcdn.apache.org/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip Unzip the resulting zip file unzip apache-livy-0.7.1-incubating-bin.zip Deploy the package and link the default symlink directory mkdir /opt/livy mv /var/tmp/apache-livy-0.7.1-incubating-bin /opt/livy/ ln -s /opt/livy/apache-livy-0.7.1-incubating-bin /opt/livy/default Step 5: Set up livy.conf Livy places template files that you'll need to copy to "real" files. You need to configure livy.conf. cp /opt/livy/default/conf/livy.conf.template /opt/livy/default/conf/livy.conf Edit the newly created livy.conf file and add two lines in the commented-out Kerberos section. livy.server.launch.kerberos.principal=${KERBEROS_PRINCIPAL} livy.server.launch.kerberos.keytab=${KERBEROS KEYTAB} Specify the full Kerberos principal name and the full path to the keytab. Step 6: Run the Livy server Livy server runs as a background process. This article doesn't discuss how to run it as a service that starts automatically.   /opt/livy/default/bin/livy-server start   Step 7: Test Livy You can use one of the recommended test commands from another node:   curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" http://<LIVY_HOST>:8998/sessions​   You can also test from a web browser:   http://<LIVY_HOST>:8998   Disclaimer: This article is contributed by an external user. The steps may not be verified by Cloudera and may not be applicable for all use cases and may be very specific to a particular distribution. Please follow with caution and at your own risk. If needed, raise a support case to get the confirmation. ... View more

I'm running a cluster with 8 worker nodes configured with 160GB Impala Daemon Memory Limit.  The worker nodes each have 370GB RAM and based on a look at the standard Host Memory Usage graph from Cloudera Manager for the nodes, it looks like I have capacity for additional query space.   My question: Does it look like I have room to increase my Impala values to meet my needs?  From my viewpoint, I think I have at least another 100GB of headroom, but I don't want to impact Hive or Spark processing that may occur during the same time windows.   I'd like to accomplish the following: I'd like to allow some queries that tend to overreach on Impala RAM additional capacity to do what they need to do.  These queries read some big tables, sometimes with thousands of partitions, and they have a tendency to run out of RAM. Reduce Impala usage of scratch directories on these large queries.  My cluster is storage-constrained, so when Impala goes heavy into the scratch directories, it not only takes a long time for the queries to finish, but the cluster's health starts to show issues. Currently, I don't have any admission control settings enabled.  Any query can use all the available resources.  I'd like to increase the available RAM for all of Impala while limiting the RAM for individual queries. Over the past week, The nodes' host memory usage graph contains the following example peaks: Peak 1:  Physical Memory Buffers: 2.6G Physical Memory Caches: 203.5G Physical Memory Capacity: 370G Physical Memory Used: 172G Swap Used: 0K Peak 2: Physical Memory Buffers: 2.6G Physical Memory Caches: 135G Physical Memory Capacity: 370G Physical Memory Used: 232G Swap Used: 768K During a quiet time, the numbers look like: Physical Memory Buffers: 70.7M Physical Memory Caches: 2.8gG Physical Memory Capacity: 370G Physical Memory Used: 19.3G Swap Used: 768K ... View more