All Cloudera Data Engineering jobs run into error with this error. This gets stuck in infinite loop to retry the operation. Please find the error stacktrace below:       22/07/14 06:30:55 INFO HadoopFSDelegationTokenProvider: Renewal interval is adjusted from 0 - 1657780255837 = -1657780255837 to 86400000 for token S3ADelegationToken/IDBroker 22/07/14 06:30:56 INFO HiveConf: Found configuration file file:/opt/hadoop/conf/hive-site.xml 22/07/14 06:30:57 WARN HiveServer2CredentialProvider: Failed to get HS2 delegation token java.util.NoSuchElementException: spark.sql.hive.hiveserver2.jdbc.url at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:246) at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:246) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.SparkConf.get(SparkConf.scala:246) at com.hortonworks.spark.deploy.yarn.security.HiveServer2CredentialProvider.obtainCredentials(HiveServer2CredentialProvider.scala:59) at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager$$anonfun$5.apply(YARNHadoopDelegationTokenManager.scala:61) at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager$$anonfun$5.apply(YARNHadoopDelegationTokenManager.scala:59) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59) at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$4.run(HadoopDelegationTokenManager.scala:254) at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$4.run(HadoopDelegationTokenManager.scala:252) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainTokensAndScheduleRenewal(HadoopDelegationTokenManager.scala:252) at org.apache.spark.deploy.security.HadoopDelegationTokenManager.org$apache$spark$deploy$security$HadoopDelegationTokenManager$$updateTokensTask(HadoopDelegationTokenManager.scala:226) at org.apache.spark.deploy.security.HadoopDelegationTokenManager.start(HadoopDelegationTokenManager.scala:128) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$$anonfun$start$1.apply(CoarseGrainedSchedulerBackend.scala:407) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$$anonfun$start$1.apply(CoarseGrainedSchedulerBackend.scala:404) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.start(CoarseGrainedSchedulerBackend.scala:404) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:46) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:191) at org.apache.spark.SparkContext.<init>(SparkContext.scala:514) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2553) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:939) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:930) at template.spark.InitSpark$class.$init$(InitSpark.scala:11) at template.spark.SimpleApp$.<init>(SimpleApp.scala:6) at template.spark.SimpleApp$.<clinit>(SimpleApp.scala) at template.spark.SimpleApp.main(SimpleApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:858) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:942) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/07/14 06:30:57 INFO HiveStreamingCredentialProvider: Obtaining delegation token (secure metastore) for hive streaming.. 22/07/14 06:30:57 INFO HiveStreamingCredentialProvider: Getting Hive delegation token for user.kk/cluster-6vfglrb7.dex.cloudera.site@PTY-CDP7.FD9S-2BVH.CLOUDERA.SITE against hive/_HOST@PTY-CDP7.FD9S-2BVH.CLOUDERA.SITE at thrift://pty-dl726-master0.pty-cdp7.fd9s-2bvh.cloudera.site:9083 22/07/14 06:30:57 INFO HiveMetaStoreClient: HMS client filtering is enabled. 22/07/14 06:30:57 INFO HiveMetaStoreClient: Trying to connect to metastore with URI thrift://pty-dl726-master0.pty-cdp7.fd9s-2bvh.cloudera.site:9083 22/07/14 06:30:57 INFO HiveMetaStoreClient: HMSC::open(): Could not find delegation token. Creating KERBEROS-based thrift connection. 22/07/14 06:30:57 INFO HiveMetaStoreClient: Opened a connection to metastore, current connections: 1 22/07/14 06:30:57 INFO HiveMetaStoreClient: Connected to metastore. 22/07/14 06:30:57 INFO HiveStreamingCredentialProvider: Added delegation token (secure metastore) for hive streaming: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 52 6b 72 69 73 68 6e 61 2e 6b 61 73 68 69 76 2f 63 6c 75 73 74 65 72 2d 36 76 66 67 6c 72 62 37 2e 64 65 78 2e 63 6c 6f 75 64 65 72 61 2e 73 69 74 65 40 50 54 59 2d 43 44 50 37 2e 46 44 39 53 2d 32 42 56 48 2e 43 4c 4f 55 44 45 52 41 2e 53 49 54 45 04 68 69 76 65 52 6b 72 69 73 68 6e 61 2e 6b 61 73 68 69 76 2f 63 6c 75 73 74 65 72 2d 36 76 66 67 6c 72 62 37 2e 64 65 78 2e 63 6c 6f 75 64 65 72 61 2e 73 69 74 65 40 50 54 59 2d 43 44 50 37 2e 46 44 39 53 2d 32 42 56 48 2e 43 4c 4f 55 44 45 52 41 2e 53 49 54 45 8a 01 81 fb 67 41 f5 8a 01 82 1f 73 c5 f5 02 01 alias: HIVE_DELEGATION_TOKEN 22/07/14 06:30:57 INFO HiveMetaStoreClient: Closed a connection to metastore, current connections: 0 22/07/14 06:30:57 INFO YARNHadoopDelegationTokenManager: Calculated delay on renewal is 64798887, based on next renewal 1657866655793 and the ratio 0.75, and current time 1657780257276 22/07/14 06:30:57 INFO YARNHadoopDelegationTokenManager: Scheduling renewal in 18.0 h. 22/07/14 06:30:57 INFO YARNHadoopDelegationTokenManager: Updating delegation tokens. 22/07/14 06:30:57 INFO SparkHadoopUtil: Updating delegation tokens for current user. 22/07/14 06:30:57 INFO SparkHadoopUtil: Updating delegation tokens for current user. 22/07/14 06:30:57 INFO Utils: Using initial executors = 2, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 22/07/14 06:30:57 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 22/07/14 06:30:58 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:30:59 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:00 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:01 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:02 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:03 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:04 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:05 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:06 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:07 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:08 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22/07/14 06:31:09 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) ... View more