Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ambari post user creation hook fail

avatar
Contributor

The issue:

I followed Administering Ambari: Enable user home directory creation to enable creating a home directory in HDFS for a user added via Ambari. However, every time I create a user, Ambari's task list shows a failure log titled "Post user creation hook for [1] users", and the content says:

stderr:   errors-2610.txt
Server action execution failed to complete!
stdout:   output-2610.txt
Server action failed

My understanding is that this is because the `admin` user has no permission to modify content of HDFS directory `/user`. `hdfs dfs -ls` commands show the following:

$ hdfs dfs -ls /
Found 13 items
drwxrwxrwt   - yarn   hadoop          0 2018-08-06 15:04 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2018-08-05 21:05 /apps
drwxr-xr-x   - yarn   hadoop          0 2018-08-02 15:34 /ats
drwxr-xr-x   - hdfs   hdfs            0 2018-08-02 15:34 /atsv2
drwxr-xr-x   - hdfs   hdfs            0 2018-08-02 15:34 /hdp
drwx------   - livy   hdfs            0 2018-08-02 15:50 /livy2-recovery
drwxr-xr-x   - mapred hdfs            0 2018-08-02 15:34 /mapred
drwxrwxrwx   - mapred hadoop          0 2018-08-02 15:35 /mr-history
drwxr-xr-x   - hdfs   hdfs            0 2018-08-02 15:34 /services
drwxrwxrwx   - spark  hadoop          0 2018-08-06 16:07 /spark2-history
drwxrwxrwx   - hdfs   hdfs            0 2018-08-05 20:53 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2018-08-06 15:46 /user
drwxr-xr-x   - hdfs   hdfs            0 2018-08-03 00:25 /warehouse
$ hdfs dfs -ls /user
Found 8 items
drwxr-xr-x   - admin     hdfs            0 2018-08-06 15:46 /user/admin
drwxrwx---   - ambari-qa hdfs            0 2018-08-05 21:00 /user/ambari-qa
drwxr-xr-x   - hive      hdfs            0 2018-08-03 19:51 /user/hive
drwxrwxr-x   - livy      hdfs            0 2018-08-02 15:50 /user/livy
drwxrwxr-x   - oozie     hdfs            0 2018-08-05 20:54 /user/oozie
drwxrwxr-x   - spark     hdfs            0 2018-08-02 15:50 /user/spark
drwxrwx---   - yarn-ats  hadoop          0 2018-08-03 19:46 /user/yarn-ats
drwxr-xr-x   - zeppelin  hdfs            0 2018-08-05 21:06 /user/zeppelin

My questions are:

Is it due to the permission problem that Ambari failed to run `post-user-creation-hook.sh`?
If yes, how do I give enough permission to the `admin` user?
If not, what might be causing the failure?

Thanks!

1 ACCEPTED SOLUTION

avatar

Hi @Guozhen Li ,

This error seems to be generic and doesn't give any clue.

Can you go into /var/lib/ambai-agent/data/errors-2610.txt and also output-2610.txt and see if you can find any clue, attach its output..

Additionally, you can do inspect /var/log/ambari-server/ambari-server.log and see what error its thrown and attach here.

Hope you have added this in core-site of hdfs

hadoop.proxyuser.root.groups=*

hadoop.proxyuser.root.hosts=*

View solution in original post

4 REPLIES 4

avatar

Hi @Guozhen Li ,

This error seems to be generic and doesn't give any clue.

Can you go into /var/lib/ambai-agent/data/errors-2610.txt and also output-2610.txt and see if you can find any clue, attach its output..

Additionally, you can do inspect /var/log/ambari-server/ambari-server.log and see what error its thrown and attach here.

Hope you have added this in core-site of hdfs

hadoop.proxyuser.root.groups=*

hadoop.proxyuser.root.hosts=*

avatar
Contributor

Thanks @Akhil S Naik.

Unfortunately I don't see a `errors-2610.txt` or `output-2610` file on the Ambari server machine in `/var/lib/ambari-agent/data` directory. There are many other errors-xxxx.txt files but not `-2610`...

But `/var/log/ambari-server/ambari-server.log` has something related to 2610:

cat ambari-server.log | grep -A 5 2610
2018-08-06 15:48:25,876 ERROR [ambari-action-scheduler] ActionScheduler:817 - Execution command has no timeout parameter{"clusterName":"citilabs_test_cluster","requestId":192,"stageId":-1,"taskId":2610,"commandId":"192--1","hostname":"_internal_ambari","role":"AMBARI_SERVER_ACTION","hostLevelParams":{},"roleParams":{"ACTION_USER_NAME":"ambari","ACTION_NAME":"org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction"},"roleCommand":"EXECUTE","clusterHostInfo":{},"configurations":{},"configurationAttributes":{},"configurationTags":{},"forceRefreshConfigTagsBeforeExecution":false,"commandParams":{"cmd-hdfs-principal":"NA","cmd-input-file":"/var/lib/ambari-server/data/tmp/user_hook_input_1533595705841.csv","cluster-security-type":"NONE","cmd-hdfs-user":"hdfs","cmd-payload":"{\"guozhen\":[]}","cmd-hdfs-keytab":"NA","hook-script":"/var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh","cluster-name":"citilabs_test_cluster","cluster-id":"2"},"serviceName":"","kerberosCommandParams":[],"localComponents":[],"availableServices":{},"componentVersionMap":{"HIVE":{"HIVE_SERVER":"3.0.0.0-1634","HIVE_SERVER_INTERACTIVE":"3.0.0.0-1634","HIVE_METASTORE":"3.0.0.0-1634","HIVE_CLIENT":"3.0.0.0-1634"},"ZEPPELIN":{"ZEPPELIN_MASTER":"3.0.0.0-1634"},"SQOOP":{"SQOOP":"3.0.0.0-1634"},"HDFS":{"SECONDARY_NAMENODE":"3.0.0.0-1634","HDFS_CLIENT":"3.0.0.0-1634","ZKFC":"3.0.0.0-1634","NFS_GATEWAY":"3.0.0.0-1634","DATANODE":"3.0.0.0-1634","JOURNALNODE":"3.0.0.0-1634","NAMENODE":"3.0.0.0-1634"},"MAPREDUCE2":{"MAPREDUCE2_CLIENT":"3.0.0.0-1634","HISTORYSERVER":"3.0.0.0-1634"},"OOZIE":{"OOZIE_CLIENT":"3.0.0.0-1634","OOZIE_SERVER":"3.0.0.0-1634"},"TEZ":{"TEZ_CLIENT":"3.0.0.0-1634"},"ZOOKEEPER":{"ZOOKEEPER_SERVER":"3.0.0.0-1634","ZOOKEEPER_CLIENT":"3.0.0.0-1634"},"SPARK2":{"SPARK2_CLIENT":"3.0.0.0-1634","SPARK2_THRIFTSERVER":"3.0.0.0-1634","LIVY2_SERVER":"3.0.0.0-1634","SPARK2_JOBHISTORYSERVER":"3.0.0.0-1634"},"YARN":{"TIMELINE_READER":"3.0.0.0-1634","NODEMANAGER":"3.0.0.0-1634","YARN_CLIENT":"3.0.0.0-1634","APP_TIMELINE_SERVER":"3.0.0.0-1634","YARN_REGISTRY_DNS":"3.0.0.0-1634","RESOURCEMANAGER":"3.0.0.0-1634"}},"commandType":"EXECUTION_COMMAND"}
2018-08-06 15:48:25,917  INFO [Server Action Executor Worker 2610] PostUserCreationHookServerAction:131 - Validating command parameters ...
2018-08-06 15:48:25,917  INFO [Server Action Executor Worker 2610] PostUserCreationHookServerAction:158 - Command parameter validation passed.
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] CsvFilePersisterService:106 - Persisting map data to csv file
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] CsvFilePersisterService:82 - Persisting collection to csv file
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] CsvFilePersisterService:86 - Collection successfully persisted to csv file.
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] ShellCommandUtilityWrapper:48 - Running command: /var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh
2018-08-06 15:48:25,923 ERROR [Server Action Executor Worker 2610] PostUserCreationHookServerAction:93 - Server action is about to quit due to an exception.
java.io.IOException: Cannot run program "/var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh": error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:457)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:513)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:526)
--
2018-08-06 15:48:25,924  WARN [Server Action Executor Worker 2610] ServerActionExecutor:471 - Task #2610 failed to complete execution due to thrown exception: org.apache.ambari.server.AmbariException:Server action execution failed to complete!
org.apache.ambari.server.AmbariException: Server action execution failed to complete!
	at org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction.execute(PostUserCreationHookServerAction.java:94)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
	at java.lang.Thread.run(Thread.java:745)

It's quite obvious now... the `ERROR` line says that `/var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh` file doesn't exist. I missed a 'c' in 'scripts' in the path. I corrected it and user home directory creation worked!

Thanks for helping me out!

My HDFS config file says these:

hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=vm-097

where `vm-097` is the Ambari server hostname. Should I be worried about this?

Finally, do you mind give some advice on how to know where to track logs when problems occur? I would have no idea where to locate the error message if it weren't for your help. (Thanks again!)

avatar

Hi @Guozhen Li

Just would like to advise you

ambari-server logs are at

/var/log/ambari-server/ambari-server.log 

ambari start up logs are (where you see out of memory, coredump if it happens )

/var/log/ambari-server/ambari-server.out

ambari-views(hive views , CapacityScheduler view ) logs are in the folder

/var/log/ambari-server/<corresponding-view-name>

ambari-agent logs are at :

/var/log/ambari-agent/ambari-agent.log

ambari operations logs (start service , stop service) will be at folder of corresponding node. here in this folder you can find corresponding error-XXXX.txt , output-XXXX.txt and command-XXXX.txt which can be used for troubleshooting

/var/lib/ambari-agent/data/

Hope this helps you.

If my answer helped you please accept the answer which would help another guys who face the same trouble.

avatar
Contributor

Best answer marked. Very helpful, thanks!