Running Spark Application on a Kerberized Hadoop cluster through Eclipse
This article explains on how to build a spark application in eclipse that can run in local mode on a kerberized hadoop cluster.
Environment:
Eclipse : Scala IDE
Requisites:
| krb5.conf | Copy /etc/krb5.conf from any one of the node of your cluster where keberos client is installed to local machine |
| core-site.xml | Copy core-site.xml from /usr/hdp/current/conf/core-site.xml to local machine |
| hdfs-site.xml | Copy hdfs-site.xml from /usr/hdp/current/conf/core-site.xml to local machine |
| yarn-site.xml | Copy yarn-site.xml from /usr/hdp/current/conf/core-site.xml to local machine |
| keytab | Copy keytab file of the user from cluster to local machine. This keytab file should be of the corresponding user principal. |
Steps:
1. Add namenode(s) hostname to host file on local machine(In case of HA, add both namenodes to host file).
2. Add hostname of KDC server to host file on local machine.
3. Right click on your project in Eclipse and add source folder, rename it to resources.
4. Copy core-site.xml, hdfs-site.xml and yarn-site.xml to resources folder(which are in local machine).
Following is the sample scala code.
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.hadoop.hdfs
import org.apache.hadoop.security.{Credentials, UserGroupInformation}
import org.apache.hadoop.security.token.{Token, TokenIdentifier}
import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier
import org.apache.spark.{SparkContext,SparkConf}
object testkerbconn {
def main(args: Array[String]){
System.setProperty("java.security.krb5.conf", "/Users/hmatta/krb5.conf")
UserGroupInformation.loginUserFromKeytab("ambari-qa-ok@EXAMPLE.COM","/Users/hmatta/smokeuser.headless.keytab")
val sparkconf = new SparkConf().setAppName("TestKerb") .setMaster("local[*]")
val sc = new SparkContext(sparkconf)
val data = sc.textFile("/tmp/idtest.ambari-qa.1502551141.07.in")
data.collect.foreach{println}
}
}
