Reply
New Contributor
Posts: 1
Registered: ‎09-07-2017

spark.eventLog.dir in spark-defaults.conf comes prefixed with wrong value on installation

After installation of cloudera on linux, while testing spark-submit on example throws below error 

 

ERROR spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/home/enablerstest/2017-09-05 17:02:21,540 INFO [main] util.KerberosName (KerberosName.java:getShortName(400)) - No auth_to_local rules applied to ADDS_linuxDNS@AVIVAAWS.COM does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598)...

......

 

On analysis found value for spark.eventLog.dir is wrong, it should 

spark.eventLog.dir=hdfs://xx-xx-xx-xx.xxxx.com:8020/user/spark/applicationHistory

instead it comes as 

spark.eventLog.dir=2017-09-05 17:02:21,540 INFO [main] util.KerberosName (KerberosName.java:getShortName(400)) - No auth_to_local rules applied to ADDS_linuxDNS@xxxx.COM
hdfs://xx-xx-xx-xx.xxxx.com:8020/user/spark/applicationHistory

---------------------------------------------------------------------------------------------------

]$ more spark-defaults.conf
spark.authenticate=false
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.executorIdleTimeout=60
spark.dynamicAllocation.minExecutors=0
spark.dynamicAllocation.schedulerBacklogTimeout=1
spark.eventLog.enabled=true
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.shuffle.service.enabled=true
spark.shuffle.service.port=7337
spark.ui.killEnabled=true
spark.eventLog.dir=2017-09-05 17:02:21,540 INFO [main] util.KerberosName (KerberosName.java:getShortName(400)) - No auth_to_local rules applied to ADDS_linuxDNS@xxxx.COM
hdfs://xx-xx-xx-xx.xxxx.com:8020/user/spark/applicationHistory
spark.yarn.historyServer.address=http://xx-xx-xx-xx.xxxx.com:18088
spark.yarn.jar=local:/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/spark/lib/spark-assembly.jar
spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/hadoop/lib/native
spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/hadoop/lib/native
spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/hadoop/lib/native
spark.yarn.config.gatewayPath=/opt/cloudera/parcels
spark.yarn.config.replacementPath={{HADOOP_COMMON_HOME}}/../../..
spark.master=yarn-client

Explorer
Posts: 14
Registered: ‎11-12-2015

Re: spark.eventLog.dir in spark-defaults.conf comes prefixed with wrong value on installation

[ Edited ]

I stumbled on the same issue. 

Actually the values gets overwritten with similar value (random exception message) occasionally. 

 

Does anyone has idea about that ?

 

 

The only thing related to spark on that node are : 

 - We have Cloudera Gateway Spark Role

 - Execute spark-submit  (usually using https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/launcher/SparkLauncher.html)

 - Spark is 1.6.1 version 

 

 

In our case the corrupted config had that value:

Spoiler
spark.eventLog.dir=2018-01-12 03:56:55,773 WARN [TGT Renewer for user@bar.foo.com] security.UserGroupInformation (UserGroupInformation.java:run(1044)) - Exception encountered while running the renewal command for user@bar.foo.com. (TGT end time:1515771836000, renewalFailures: org.apache.hadoop.metrics2.lib.MutableGaugeInt@75ec7517,renewalFailuresTotal: org.apache.hadoop.metrics2.lib.MutableGaugeLong@30eef5ce)
ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials

at org.apache.hadoop.util.Shell.runcommand(Shell.java:601)
at org.apache.hadoop.util.Shell.run(Shell.java:504)
at org.apache.hadoop.util.Shell$ShellcommandExecutor.execute(Shell.java:786)
at org.apache.hadoop.util.Shell.execcommand(Shell.java:879)
at org.apache.hadoop.util.Shell.execcommand(Shell.java:862)
at org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:1023)
at java.lang.Thread.run(Thread.java:745)
hdfs://node:8020/user/spark/applicationHistory

 

Announcements