question Re: Ambari post user creation hook fail in Support Questions

Ambari post user creation hook fail

ligzligz — Tue, 07 Aug 2018 06:19:19 GMT

The issue:

I followed Administering Ambari: Enable user home directory creation to enable creating a home directory in HDFS for a user added via Ambari. However, every time I create a user, Ambari's task list shows a failure log titled "Post user creation hook for [1] users", and the content says:

stderr:   errors-2610.txt
Server action execution failed to complete!
stdout:   output-2610.txt
Server action failed

My understanding is that this is because the `admin` user has no permission to modify content of HDFS directory `/user`. `hdfs dfs -ls` commands show the following:

$ hdfs dfs -ls /
Found 13 items
drwxrwxrwt   - yarn   hadoop          0 2018-08-06 15:04 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2018-08-05 21:05 /apps
drwxr-xr-x   - yarn   hadoop          0 2018-08-02 15:34 /ats
drwxr-xr-x   - hdfs   hdfs            0 2018-08-02 15:34 /atsv2
drwxr-xr-x   - hdfs   hdfs            0 2018-08-02 15:34 /hdp
drwx------   - livy   hdfs            0 2018-08-02 15:50 /livy2-recovery
drwxr-xr-x   - mapred hdfs            0 2018-08-02 15:34 /mapred
drwxrwxrwx   - mapred hadoop          0 2018-08-02 15:35 /mr-history
drwxr-xr-x   - hdfs   hdfs            0 2018-08-02 15:34 /services
drwxrwxrwx   - spark  hadoop          0 2018-08-06 16:07 /spark2-history
drwxrwxrwx   - hdfs   hdfs            0 2018-08-05 20:53 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2018-08-06 15:46 /user
drwxr-xr-x   - hdfs   hdfs            0 2018-08-03 00:25 /warehouse

$ hdfs dfs -ls /user
Found 8 items
drwxr-xr-x   - admin     hdfs            0 2018-08-06 15:46 /user/admin
drwxrwx---   - ambari-qa hdfs            0 2018-08-05 21:00 /user/ambari-qa
drwxr-xr-x   - hive      hdfs            0 2018-08-03 19:51 /user/hive
drwxrwxr-x   - livy      hdfs            0 2018-08-02 15:50 /user/livy
drwxrwxr-x   - oozie     hdfs            0 2018-08-05 20:54 /user/oozie
drwxrwxr-x   - spark     hdfs            0 2018-08-02 15:50 /user/spark
drwxrwx---   - yarn-ats  hadoop          0 2018-08-03 19:46 /user/yarn-ats
drwxr-xr-x   - zeppelin  hdfs            0 2018-08-05 21:06 /user/zeppelin

My questions are:

Is it due to the permission problem that Ambari failed to run `post-user-creation-hook.sh`?
If yes, how do I give enough permission to the `admin` user?
If not, what might be causing the failure?

Thanks!

Re: Ambari post user creation hook fail

akhilsnaik — Tue, 07 Aug 2018 09:05:39 GMT

Hi @Guozhen Li ,

This error seems to be generic and doesn't give any clue.

Can you go into /var/lib/ambai-agent/data/errors-2610.txt and also output-2610.txt and see if you can find any clue, attach its output..

Additionally, you can do inspect /var/log/ambari-server/ambari-server.log and see what error its thrown and attach here.

Hope you have added this in core-site of hdfs

hadoop.proxyuser.root.groups=*

hadoop.proxyuser.root.hosts=*

Re: Ambari post user creation hook fail

ligzligz — Tue, 07 Aug 2018 12:06:07 GMT

Thanks @Akhil S Naik.

Unfortunately I don't see a `errors-2610.txt` or `output-2610` file on the Ambari server machine in `/var/lib/ambari-agent/data` directory. There are many other errors-xxxx.txt files but not `-2610`...

But `/var/log/ambari-server/ambari-server.log` has something related to 2610:

cat ambari-server.log | grep -A 5 2610
2018-08-06 15:48:25,876 ERROR [ambari-action-scheduler] ActionScheduler:817 - Execution command has no timeout parameter{"clusterName":"citilabs_test_cluster","requestId":192,"stageId":-1,"taskId":2610,"commandId":"192--1","hostname":"_internal_ambari","role":"AMBARI_SERVER_ACTION","hostLevelParams":{},"roleParams":{"ACTION_USER_NAME":"ambari","ACTION_NAME":"org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction"},"roleCommand":"EXECUTE","clusterHostInfo":{},"configurations":{},"configurationAttributes":{},"configurationTags":{},"forceRefreshConfigTagsBeforeExecution":false,"commandParams":{"cmd-hdfs-principal":"NA","cmd-input-file":"/var/lib/ambari-server/data/tmp/user_hook_input_1533595705841.csv","cluster-security-type":"NONE","cmd-hdfs-user":"hdfs","cmd-payload":"{\"guozhen\":[]}","cmd-hdfs-keytab":"NA","hook-script":"/var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh","cluster-name":"citilabs_test_cluster","cluster-id":"2"},"serviceName":"","kerberosCommandParams":[],"localComponents":[],"availableServices":{},"componentVersionMap":{"HIVE":{"HIVE_SERVER":"3.0.0.0-1634","HIVE_SERVER_INTERACTIVE":"3.0.0.0-1634","HIVE_METASTORE":"3.0.0.0-1634","HIVE_CLIENT":"3.0.0.0-1634"},"ZEPPELIN":{"ZEPPELIN_MASTER":"3.0.0.0-1634"},"SQOOP":{"SQOOP":"3.0.0.0-1634"},"HDFS":{"SECONDARY_NAMENODE":"3.0.0.0-1634","HDFS_CLIENT":"3.0.0.0-1634","ZKFC":"3.0.0.0-1634","NFS_GATEWAY":"3.0.0.0-1634","DATANODE":"3.0.0.0-1634","JOURNALNODE":"3.0.0.0-1634","NAMENODE":"3.0.0.0-1634"},"MAPREDUCE2":{"MAPREDUCE2_CLIENT":"3.0.0.0-1634","HISTORYSERVER":"3.0.0.0-1634"},"OOZIE":{"OOZIE_CLIENT":"3.0.0.0-1634","OOZIE_SERVER":"3.0.0.0-1634"},"TEZ":{"TEZ_CLIENT":"3.0.0.0-1634"},"ZOOKEEPER":{"ZOOKEEPER_SERVER":"3.0.0.0-1634","ZOOKEEPER_CLIENT":"3.0.0.0-1634"},"SPARK2":{"SPARK2_CLIENT":"3.0.0.0-1634","SPARK2_THRIFTSERVER":"3.0.0.0-1634","LIVY2_SERVER":"3.0.0.0-1634","SPARK2_JOBHISTORYSERVER":"3.0.0.0-1634"},"YARN":{"TIMELINE_READER":"3.0.0.0-1634","NODEMANAGER":"3.0.0.0-1634","YARN_CLIENT":"3.0.0.0-1634","APP_TIMELINE_SERVER":"3.0.0.0-1634","YARN_REGISTRY_DNS":"3.0.0.0-1634","RESOURCEMANAGER":"3.0.0.0-1634"}},"commandType":"EXECUTION_COMMAND"}
2018-08-06 15:48:25,917  INFO [Server Action Executor Worker 2610] PostUserCreationHookServerAction:131 - Validating command parameters ...
2018-08-06 15:48:25,917  INFO [Server Action Executor Worker 2610] PostUserCreationHookServerAction:158 - Command parameter validation passed.
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] CsvFilePersisterService:106 - Persisting map data to csv file
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] CsvFilePersisterService:82 - Persisting collection to csv file
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] CsvFilePersisterService:86 - Collection successfully persisted to csv file.
2018-08-06 15:48:25,919  INFO [Server Action Executor Worker 2610] ShellCommandUtilityWrapper:48 - Running command: /var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh
2018-08-06 15:48:25,923 ERROR [Server Action Executor Worker 2610] PostUserCreationHookServerAction:93 - Server action is about to quit due to an exception.
java.io.IOException: Cannot run program "/var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh": error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:457)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:513)
	at org.apache.ambari.server.utils.ShellCommandUtil.runCommand(ShellCommandUtil.java:526)
--
2018-08-06 15:48:25,924  WARN [Server Action Executor Worker 2610] ServerActionExecutor:471 - Task #2610 failed to complete execution due to thrown exception: org.apache.ambari.server.AmbariException:Server action execution failed to complete!
org.apache.ambari.server.AmbariException: Server action execution failed to complete!
	at org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction.execute(PostUserCreationHookServerAction.java:94)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
	at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
	at java.lang.Thread.run(Thread.java:745)

It's quite obvious now... the `ERROR` line says that `/var/lib/ambari-server/resources/sripts/post-user-creation-hook.sh` file doesn't exist. I missed a 'c' in 'scripts' in the path. I corrected it and user home directory creation worked!

Thanks for helping me out!

My HDFS config file says these:

hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=vm-097

where `vm-097` is the Ambari server hostname. Should I be worried about this?

Finally, do you mind give some advice on how to know where to track logs when problems occur? I would have no idea where to locate the error message if it weren't for your help. (Thanks again!)

Re: Ambari post user creation hook fail

akhilsnaik — Tue, 07 Aug 2018 12:18:09 GMT

Hi @Guozhen Li

Just would like to advise you

ambari-server logs are at

/var/log/ambari-server/ambari-server.log

ambari start up logs are (where you see out of memory, coredump if it happens )

/var/log/ambari-server/ambari-server.out

ambari-views(hive views , CapacityScheduler view ) logs are in the folder

/var/log/ambari-server/<corresponding-view-name>

ambari-agent logs are at :

/var/log/ambari-agent/ambari-agent.log

ambari operations logs (start service , stop service) will be at folder of corresponding node. here in this folder you can find corresponding error-XXXX.txt , output-XXXX.txt and command-XXXX.txt which can be used for troubleshooting

/var/lib/ambari-agent/data/

Hope this helps you.

If my answer helped you please accept the answer which would help another guys who face the same trouble.

Re: Ambari post user creation hook fail

ligzligz — Tue, 07 Aug 2018 12:40:46 GMT

Best answer marked. Very helpful, thanks!