question Re: Restart Resource Manager does not work, can not start YARN in Archives of Support Questions (Read Only)

Restart Resource Manager does not work, can not start YARN

kuehnert_delitz — Mon, 19 Nov 2018 18:53:25 GMT

I set up successfully the Ambari server but some sevices can not be started.

If I want o start or restart the resource manager it aborts (after some time) giving me following message:

2018-11-19 00:13:30,067 - get_user_call_output returned (0, u'{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File does not exist: /ats/done/"}}404', u'')
2018-11-19 00:13:30,068 - Will retry 2 time(s), caught exception: DFS directory '/ats/done/' does not exist !. Sleeping for 20 sec(s)

It says /ats/done is not existing. Where do I create this folder?

Re: Restart Resource Manager does not work, can not start YARN

asirna — Mon, 19 Nov 2018 23:24:04 GMT

@Andreas Kühnert,

I guess the directory got deleted somehow from HDFS. You can create it and try starting RM

# su hdfs
# kinit -kt /etc/security/keytabs/hdfs.headless.keytab {principal}  ---> Run this if your environment is kerberized
# hdfs dfs -mkdir -p /ats/done
# hdfs dfs -chown -R yarn:hadoop /ats/done

After running the commands, try to restart RM. If it fails with /ats/active directory not found, repeat the same steps changing the directory name.

If this works , please take a moment to login and "Accept" the answer.

Re: Restart Resource Manager does not work, can not start YARN

kuehnert_delitz — Wed, 21 Nov 2018 07:23:51 GMT

Hello it worked.

But now I am facing another issue. When I am starting YARN Registry DNS it says:

 File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ulimit -c unlimited; export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.0.0-1634/hadoop/libexec && /usr/hdp/3.0.0.0-1634/hadoop-yarn/bin/yarn --config /usr/hdp/3.0.0.0-1634/hadoop/conf --daemon start registrydns' returned 1. mesg: ttyname failed: Inappropriate ioctl for device
ERROR: Cannot set priority of registrydns process 24309

I already set ulimit to 10000 as recommended.

Whats the issue here?

Re: Restart Resource Manager does not work, can not start YARN

kuehnert_delitz — Wed, 21 Nov 2018 07:23:53 GMT

Okay it worked!

But now I am facing issue with starting YARN Registry DNS.

 File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ulimit -c unlimited; export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.0.0-1634/hadoop/libexec && /usr/hdp/3.0.0.0-1634/hadoop-yarn/bin/yarn --config /usr/hdp/3.0.0.0-1634/hadoop/conf --daemon start registrydns' returned 1. mesg: ttyname failed: Inappropriate ioctl for device
ERROR: Cannot set priority of registrydns process 24309

Whats the issue here? Why it can not set its priority?

Thanks in advance 😉

Re: Restart Resource Manager does not work, can not start YARN

asirna — Wed, 21 Nov 2018 11:46:55 GMT

@Andreas Kühnert,

Glad that the issue is resolved. Since this is a different issue, I suggest to open a new thread for this issue so that the main thread doesn't get deviated. I'm not sure of that issue, may be other experts can help 🙂