Using Rhive with Kerberized Hadoop Cluster

[#1]

Goal

• Access Hive through R shell with Rhive
• Authentication mechanism: Kerberos

Assumptions

• Have a HDP cluster up and running
• Have configured and installed R
• Cluster is configured on CentOS7
• Need to have openssl-devel installed on the server where R is installed
• If using openjdk, ensure that the JAVA_HOME is set to the path where javac is available

Steps

• Download Rserve

wget https://rforge.net/Rserve/snapshot/Rserve_1.8-5.tar.gz

• Use R to install Rserve

R CMD INSTALL Rserve_1.8-5.tar.gz

• Download rhive library

git clone https://github.com/nexr/RHive.gita

• Set the appropriate path for HIVE_HOME and HADOOP_HOME

export HADOOP_HOME=/usr/hdp/2.5.0.0-1245/hadoop; export HIVE_HOME=/usr/hdp/2.5.0.0-1245/hivea

• Change directory to RHive

cd RHive

• Use ant to build

ant build

• Use R to build RHive, this will build a file within the same directory

R CMD build RHive

• Use the newly created file name to install RHive library

R CMD INSTALL RHive_2.0-0.10.tar.gz

Demo

• Launch R using "R" at the unix prompt
• Load RHive library

> library(RHive)
Loading required package: rJava
Loading required package: Rserve

• Use the connection string in this format

> rhive.connect(host="node1.hortonworks.com:10000/default;principal=hive/[email protected];AuthMech=1;KrbHostFQDN=service.hortonworks.com;KrbServiceName=hive;KrbRealm=HDP.COM", defaultFS="hdfs://node1.hortonworks.com/rhive", hiveServer2=TRUE,updateJar=FALSE)

2016-11-09 22:30:00,425 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-11-09 22:30:01,361 WARN  [main] shortcircuit.DomainSocketFactory (DomainSocketFactory.java:<init>(117)) - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2016-11-09 22:30:01,853 INFO  [Thread-7] jdbc.Utils (Utils.java:parseURL(316)) - Supplied authorities: node1.hortonworks.com:10000
2016-11-09 22:30:01,853 INFO  [Thread-7] jdbc.Utils (Utils.java:parseURL(432)) - Resolved authority: node1.hortonworks.com:10000

• Querying a table

> rhive.query("SELECT * FROM default.test")
  test.col1 test.col2
1         1         1
2         2         2
3         3         3
4         4         4
> 

Enjoy !!

[#2]

Where are steps to configure R-server for Kerberos? Does it not need any kerberos related settings? No kinit etc..?

I noticed that the connect string has kerberos related variables though. Is it really good enough? Please confirm.

rhive.connect(host="node1.hortonworks.com:10000/default;principal=hive/[email protected];AuthMech=1;KrbHostFQDN=service.hortonworks.com;KrbServiceName=hive;KrbRealm=HDP.COM", defaultFS="hdfs://node1.hortonworks.com/rhive", hiveServer2=TRUE,updateJar=FALSE))