- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Spark in Yarn mode gives error when installed in kerberized cluster
Created on 01-22-2015 07:59 AM - edited 09-16-2022 02:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi ,
I am using CDH 5.2 on RHEL 6.5
I am trying to install spark in yarn mode in kerberized environment.
But it fails on the 3 rd step when it tries to upload the jars after creating history server and user dir.
+ echo 'Using /var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand as conf dir'
+ echo 'Using scripts/control.sh as process script'
+ export COMMON_SCRIPT=/usr/lib64/cmf/service/common/cloudera-config.sh
+ COMMON_SCRIPT=/usr/lib64/cmf/service/common/cloudera-config.sh
+ chmod u+x /var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/scripts/control.sh
+ exec /var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/scripts/control.sh upload_jar
Thu Jan 22 10:41:50 EST 2015
Thu Jan 22 10:41:50 EST 2015: Detected CDH_VERSION of [5]
Thu Jan 22 10:41:50 EST 2015: Uploading Spark assembly jar to '/user/spark/share/lib/spark-assembly.jar' on CDH 5 cluster
+ export SCM_KERBEROS_PRINCIPAL=spark/itsusmpl00512.xxx.com@CDH5.xxx.COM
+ SCM_KERBEROS_PRINCIPAL=spark/itsusmpl00512.xxx.com@CDH5.xxx.COM
+ acquire_kerberos_tgt spark_on_yarn.keytab
+ '[' -z spark_on_yarn.keytab ']'
+ '[' -n spark/itsusmpl00512.xxx.com@CDH5.xxx.COM ']'
+ '[' -d /usr/kerberos/bin ']'
+ which kinit
+ '[' 0 -ne 0 ']'
++ id -u
+ export KRB5CCNAME=/var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/krb5cc_481
+ KRB5CCNAME=/var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/krb5cc_481
+ echo 'using spark/itsusmpl00512.jnj.com@CDH5.JNJ.COM as Kerberos principal'
+ echo 'using /var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/krb5cc_481 as Kerberos ticket cache'
+ kinit -c /var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/krb5cc_481 -kt /var/run/cloudera-scm-agent/process/295-spark_on_yarn-SPARK_YARN_HISTORY_SERVER-SparkUploadJarCommand/spark_on_yarn.keytab spark/itsusmpl00512.xxx.com@CDH5.xxx.COM
kinit: Cannot resolve network address for KDC in realm "CDH5.xxx.COM" while getting initial credentials
+ '[' 1 -ne 0 ']'
+ echo 'kinit was not successful.'
+ exit 1
Created 02-03-2015 01:24 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I was able to resolve this,
I changed a property in krb5.conf
dns_lookup_kdc=true
I was getting an error in creating KDC initial credentials.By changing this property from false to true i was able to install spark on secure kerberized with sentry cluster.
Created 02-03-2015 01:24 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I was able to resolve this,
I changed a property in krb5.conf
dns_lookup_kdc=true
I was getting an error in creating KDC initial credentials.By changing this property from false to true i was able to install spark on secure kerberized with sentry cluster.
Created 02-05-2015 03:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you for letting us know the solution. 🙂