Capacity scheduler - error while executing job

[#1]

Hi. I try using YARN Queue Manager but I've got an issue.

I created group 'developers' (groupadd command), then I created user 'mgrabowski' which is a member of that group. Then I executed command 'build/env/bin/hue useradmin_sync_with_unix'. In YARN Queue Manager I created queue 'q_developers' and I set the parameter 'Queue Mappings' on 'g:developers:q_developers'.

When I try executing Hive query in Hue I've got an error.

 0, text: $root.design.watch.logs().join('\n')">ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1469000725460_0018 to YARN : Failed to submit application application_1469000725460_0018 submitted by user mgrabowski reason: No groups found for user mgrabowski
	at org.apache.tez.client.TezClient.start(TezClient.java:413)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
	at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
	at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1469000725460_0018 to YARN : Failed to submit application application_1469000725460_0018 submitted by user mgrabowski reason: No groups found for user mgrabowski
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
	at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
	at org.apache.tez.client.TezClient.start(TezClient.java:408)
	... 23 more

Does anyone know why? It is weird that 'u:mgrabowski:developers' works fine.

[#2]

It is throwing error for user mgrabowski

Have you allowed this user to submit jobs to yarn or is this user part of allowed groups?

[#3]

This user is a 'developers' group member too.

When Queue Mappings equals 'u:matgrabowski:q_developers' everything works fine. But 'g:developers:q_developers' doesn't work.

[#4]

I heard there is some group caching in HDFS. But it should be refreshed after 5 minutes

hadoop.security.groups.cache.secs

Any chance to restart hdfs/yarn to make sure thats not the problem?

[#5 · Accepted solution]

Ok, I know what was the problem. I've got 3-node cluster, but I created user accounts only on Hue node (node 3). When I created user accounts on node 1 and 2, everything seems good.

But I'm not sure if it's a good solution. I have to create user accounts and groups on every node in my cluster? What if I will have 1000 nodes?

[#6]

@Mateusz : In that case you may need to write a small shell script to create user or group or add user to a group since we will have ssh keys enabled. Else you may need to try expect command which will pass the password too. Hope it clarifies. If you are satisfied pls rate.

[#7]

On big clusters people normally setup an ldap server. Ipa for example is free and simple. Look on github for the security workshops of Ali baijwa. Or as said below use a ssh script or ansible or pshell to run commands on all nodes. Note some more esoteric components of the stack require that usernames have the same uid on all nodes of the cluster.

https://github.com/abajwa-hw

[#8]

Can you tell me which of components require that usernames have the same uid on all nodes and why? It's interesting.

[#9]

Honestly if I knew I would have mentioned them :-). I setup a cluster with a simple shell script

ssh-all.sh:

for i in server1 server2 server3; do ssh $i S1; done

and created users manually on a small cluster before ( we only had ~10 users so it didn't seem worth it to setup LDAP ). I never bothered about uids and never ran into problems. But we used standard stuff oozie, hive ... and never ran into problems. But other people told me that some components don't take this well.

Honestly not sure which could be that I am sure that Namenode HA setup with NFS does not work because NFS depends on the same UID but I have problems thinking of a component that would need the same uids in an hadoop environment. HDFS does not care about uids. It cares about usernames.

[#10]

I had the same problem and I also solve it adding the user to all controler nodes.

Run a script for add them from on_to_all

______________________________________________________________________________________________

#!/bin/bash
# Linux/UNIX box with ssh key based login
SERVERS=/root/hadoop_hosts
# SSH User name
USR="root"
# Email
SUBJECT="Server user login report"
EMAIL="your_e-mail@here"
EMAILMESSAGE="/tmp/sshpool_`date +%Y%m%d-%H:%M`.txt"
# create new file
>$EMAILMESSAGE
# connect each host and pull up user listing
for host in `cat $SERVERS`
do
echo "--------------------------------" >>$EMAILMESSAGE
echo "* HOST: $host " >>$EMAILMESSAGE
echo "--------------------------------" >>$EMAILMESSAGE
###ssh $USR@$host w >> $EMAILMESSAGE
ssh -tq -o "BatchMode yes" $USR@$host $1 >> $EMAILMESSAGE
done
# send an email using /bin/mail
######/bin/mailx -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<"
echo ">>> check the output file " $EMAILMESSAGE

_________________________________________________________________________________________

put DNS servers names into /root/hadoop_hosts

Also in linux there is a good command called pssh to run comands in parallel in computer clusters

🙂