Support Questions
Find answers, ask questions, and share your expertise

HDP - spark jobs failing in cluster mode

Explorer

Team,

We have spark jobs running in cluster mode and using Hiveserver2 , wherein the data is ingested from a tool and then loaded to hive.

 

the below error is observed while connecting to hive and its shown intermittently.

We checked hive server is working normal with no issues.

Let us know if any parameter to be added in spark-submit command for below resolution

 

*******************************************************

spark-submit command

/usr/hdp/current/spark2-client/bin/spark-submit --master yarn --queue udif --driver-memory ${driver_memory} --num-executors ${num_executors} --executor-memory ${executor_memory} --executor-cores 4 --conf spark.port.maxRetries=50 --conf spark.network.timeout=600s --conf spark.executor.heartbeatInterval=200s --class custom-class --conf spark.security.credentials.hiveserver2.enabled=true --conf spark.sql.hive.hiveserver2.jdbc.url="hive-jdbc string" --jars custom-jars

 

***********************************************

 

Error

***********************************************

Caused by: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: Could not establish connection to jdbc:hive2://hive-host:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=delegationToken: HTTP Response code: 401
at shadehive.org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:344)
at shadehive.org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:53)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:291)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:883)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:436)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:365)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
at com.hortonworks.spark.sql.hive.llap.JDBCWrapper.getConnector(HS2JDBCWrapper.scala:424)
at com.hortonworks.spark.sql.hive.llap.JDBCWrapper.getConnector(HS2JDBCWrapper.scala:453)
at com.hortonworks.spark.sql.hive.llap.DefaultJDBCWrapper.getConnector(HS2JDBCWrapper.scala)
at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.lambda$new$0(HiveWarehouseSessionImpl.java:85)
at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeUpdate(HiveWarehouseSessionImpl.java:205)
***********************************************************