https://community.cloudera.com/t5/Support-Questions/CDP-spark-shell-master-yarn-security/m-p/349117#M235531 <P>Hi&nbsp;<a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/53847">@paulo_klein</a>,&nbsp; Apache Spark by default request a delegation token for all 4 services: HDFS, YARN, Hive, and HBase. It is printed as a WARN message but it does no harm. It is captured due to the fact that no HBase jars are in the Spark classpath, hence, it's unable to get the HBase&nbsp;<SPAN>DelegationToken.</SPAN></P><P>&nbsp;</P><P>To fix this issue you start spark-shell or pyspark or spark-submit via<BR />--conf spark.security.credentials.hbase.enabled=false<BR /><BR />Example: # spark-shell --conf spark.security.credentials.hbase.enabled=false</P> Sat, 30 Jul 2022 01:51:31 GMT jagadeesan 2022-07-30T01:51:31Z https://community.cloudera.com/t5/Support-Questions/CDP-spark-shell-master-yarn-security/m-p/349105#M235524 <P>When I try to start a spark-shell with YARN I'm getting this error:</P><P>&nbsp;</P><P>22/07/29 15:36:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<BR />22/07/29 15:36:08 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.<BR /><FONT color="#FF6600"><STRONG>22/07/29 15:36:09 WARN security.HBaseDelegationTokenProvider: Fail to invoke HBaseConfiguration</STRONG></FONT><BR /><FONT color="#FF6600"><STRONG>java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration</STRONG></FONT><BR />at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)<BR />at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)<BR />at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)<BR />at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.hbaseConf(HBaseDelegationTokenProvider.scala:117)<BR />at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.delegationTokensRequired(HBaseDelegationTokenProvider.scala:110)<BR />at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$6.apply(HadoopDelegationTokenManager.scala:165)<BR />at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$6.apply(HadoopDelegationTokenManager.scala:164)</P><P>&nbsp;</P><P>How can I fix this issue?</P><P>&nbsp;</P><P>&nbsp;</P> Fri, 29 Jul 2022 18:39:51 GMT https://community.cloudera.com/t5/Support-Questions/CDP-spark-shell-master-yarn-security/m-p/349105#M235524 paulo_klein 2022-07-29T18:39:51Z https://community.cloudera.com/t5/Support-Questions/CDP-spark-shell-master-yarn-security/m-p/349117#M235531 <P>Hi&nbsp;<a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/53847">@paulo_klein</a>,&nbsp; Apache Spark by default request a delegation token for all 4 services: HDFS, YARN, Hive, and HBase. It is printed as a WARN message but it does no harm. It is captured due to the fact that no HBase jars are in the Spark classpath, hence, it's unable to get the HBase&nbsp;<SPAN>DelegationToken.</SPAN></P><P>&nbsp;</P><P>To fix this issue you start spark-shell or pyspark or spark-submit via<BR />--conf spark.security.credentials.hbase.enabled=false<BR /><BR />Example: # spark-shell --conf spark.security.credentials.hbase.enabled=false</P> Sat, 30 Jul 2022 01:51:31 GMT https://community.cloudera.com/t5/Support-Questions/CDP-spark-shell-master-yarn-security/m-p/349117#M235531 jagadeesan 2022-07-30T01:51:31Z