Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark on Yarn fails with LDAP

Highlighted

Spark on Yarn fails with LDAP

Rising Star

Hello,

 

I have configured CDH cluster with LDAP integration and CompositeGroupMapping (ShellBasedUnixGroupsMapping and LdapGroupsMapping) on HDFS. HDFS, Hive and Impala works great with both local user principals as well as AD users.

 

The problem I have now is with Spark (on YARN), where jobs submitted by local users work, but those submitted by AD users fail:

 

main : run as user is ldap1
main : requested yarn user is ldap1
User ldap1 not found

 

If I create user ldap1 on all hosts, then Spark works. What am I missing here?

 

Thank you

1 REPLY 1

Re: Spark on Yarn fails with LDAP

Master Guru
YARN in secure mode requires locally available user accounts to fully isolate the task containers: https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_other_hadoop_security.html#to...

You'll need to make these accounts visible to your Linux hosts via SSSD or similar software.