All CDE jobs run into error with issues in HiveServer2 delegation token

[#1]

All Cloudera Data Engineering jobs run into error with this error. This gets stuck in infinite loop to retry the operation. Please find the error stacktrace below:

22/07/14 06:30:55 INFO HadoopFSDelegationTokenProvider: Renewal interval is adjusted from 0 - 1657780255837 = -1657780255837 to 86400000 for token S3ADelegationToken/IDBroker
22/07/14 06:30:56 INFO HiveConf: Found configuration file file:/opt/hadoop/conf/hive-site.xml
22/07/14 06:30:57 WARN HiveServer2CredentialProvider: Failed to get HS2 delegation token
java.util.NoSuchElementException: spark.sql.hive.hiveserver2.jdbc.url
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:246)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:246)
at com.hortonworks.spark.deploy.yarn.security.HiveServer2CredentialProvider.obtainCredentials(HiveServer2CredentialProvider.scala:59)
at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager$$anonfun$5.apply(YARNHadoopDelegationTokenManager.scala:61)
at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager$$anonfun$5.apply(YARNHadoopDelegationTokenManager.scala:59)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59)
at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$4.run(HadoopDelegationTokenManager.scala:254)
at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$4.run(HadoopDelegationTokenManager.scala:252)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainTokensAndScheduleRenewal(HadoopDelegationTokenManager.scala:252)
at org.apache.spark.deploy.security.HadoopDelegationTokenManager.org$apache$spark$deploy$security$HadoopDelegationTokenManager$$updateTokensTask(HadoopDelegationTokenManager.scala:226)
at org.apache.spark.deploy.security.HadoopDelegationTokenManager.start(HadoopDelegationTokenManager.scala:128)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$$anonfun$start$1.apply(CoarseGrainedSchedulerBackend.scala:407)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$$anonfun$start$1.apply(CoarseGrainedSchedulerBackend.scala:404)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.start(CoarseGrainedSchedulerBackend.scala:404)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:46)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:514)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2553)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:939)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:930)
at template.spark.InitSpark$class.$init$(InitSpark.scala:11)
at template.spark.SimpleApp$.<init>(SimpleApp.scala:6)
at template.spark.SimpleApp$.<clinit>(SimpleApp.scala)
at template.spark.SimpleApp.main(SimpleApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:858)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:942)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/07/14 06:30:57 INFO HiveStreamingCredentialProvider: Obtaining delegation token (secure metastore) for hive streaming..
22/07/14 06:30:57 INFO HiveStreamingCredentialProvider: Getting Hive delegation token for user.kk/cluster-6vfglrb7.dex.cloudera.site@PTY-CDP7.FD9S-2BVH.CLOUDERA.SITE against hive/_HOST@PTY-CDP7.FD9S-2BVH.CLOUDERA.SITE at thrift://pty-dl726-master0.pty-cdp7.fd9s-2bvh.cloudera.site:9083
22/07/14 06:30:57 INFO HiveMetaStoreClient: HMS client filtering is enabled.
22/07/14 06:30:57 INFO HiveMetaStoreClient: Trying to connect to metastore with URI thrift://pty-dl726-master0.pty-cdp7.fd9s-2bvh.cloudera.site:9083
22/07/14 06:30:57 INFO HiveMetaStoreClient: HMSC::open(): Could not find delegation token. Creating KERBEROS-based thrift connection.
22/07/14 06:30:57 INFO HiveMetaStoreClient: Opened a connection to metastore, current connections: 1
22/07/14 06:30:57 INFO HiveMetaStoreClient: Connected to metastore.
22/07/14 06:30:57 INFO HiveStreamingCredentialProvider: Added delegation token (secure metastore) for hive streaming: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 52 6b 72 69 73 68 6e 61 2e 6b 61 73 68 69 76 2f 63 6c 75 73 74 65 72 2d 36 76 66 67 6c 72 62 37 2e 64 65 78 2e 63 6c 6f 75 64 65 72 61 2e 73 69 74 65 40 50 54 59 2d 43 44 50 37 2e 46 44 39 53 2d 32 42 56 48 2e 43 4c 4f 55 44 45 52 41 2e 53 49 54 45 04 68 69 76 65 52 6b 72 69 73 68 6e 61 2e 6b 61 73 68 69 76 2f 63 6c 75 73 74 65 72 2d 36 76 66 67 6c 72 62 37 2e 64 65 78 2e 63 6c 6f 75 64 65 72 61 2e 73 69 74 65 40 50 54 59 2d 43 44 50 37 2e 46 44 39 53 2d 32 42 56 48 2e 43 4c 4f 55 44 45 52 41 2e 53 49 54 45 8a 01 81 fb 67 41 f5 8a 01 82 1f 73 c5 f5 02 01 alias: HIVE_DELEGATION_TOKEN
22/07/14 06:30:57 INFO HiveMetaStoreClient: Closed a connection to metastore, current connections: 0
22/07/14 06:30:57 INFO YARNHadoopDelegationTokenManager: Calculated delay on renewal is 64798887, based on next renewal 1657866655793 and the ratio 0.75, and current time 1657780257276
22/07/14 06:30:57 INFO YARNHadoopDelegationTokenManager: Scheduling renewal in 18.0 h.
22/07/14 06:30:57 INFO YARNHadoopDelegationTokenManager: Updating delegation tokens.
22/07/14 06:30:57 INFO SparkHadoopUtil: Updating delegation tokens for current user.
22/07/14 06:30:57 INFO SparkHadoopUtil: Updating delegation tokens for current user.
22/07/14 06:30:57 INFO Utils: Using initial executors = 2, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
22/07/14 06:30:57 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
22/07/14 06:30:58 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:30:59 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:00 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:01 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:02 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:03 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:04 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:05 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:06 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:07 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:08 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
22/07/14 06:31:09 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)