Cluster Installation failed

[#1]

When I try to re-install cluster via Ambari it failed with the following error.

parent directory /usr/hdp/current/slider-client/lib doesn't exist

Checking /usr/hdp/current/ and find two directories that are not symlink

drwxr-xr-x 3 root root 17 Feb 23 21:39 oozie-client
drwxr-xr-x 3 root root 17 Feb 23 22:39 slider-client

[#2]

@Zaili Xu,

You can try installing the client using REST API and retry

1) Install Oozie client

curl -k -u {ambari-user-name}:{ambari-password} -H "X-Requested-By:ambari" -i -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://{ambari-host}:{ambari-port}/api/v1/clusters/{clustername}/hosts/{hostname}/host_components/OO... 

Note: If the above curl call gives 404, then run step 2 and re-run step 1

.

2) Add Oozie client

curl -k -u {ambari-user-name}:{ambari-password} -H "X-Requested-By:ambari" -X POST http://{ambari-host}:{ambari-port}/api/v1/clusters/{clustername}/hosts/{hostname}/host_components/OO... 

.

3) Install Slider client

curl -k -u {ambari-user-name}:{ambari-password} -H "X-Requested-By:ambari" -i -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://{ambari-host}:{ambari-port}/api/v1/clusters/{clustername}/hosts/{hostname}/host_components/SL... 

Note: If the above curl call gives 404, then run step 4 and re-run step 3

.

4) Add Slider client

curl -k -u {ambari-user-name}:{ambari-password} -H "X-Requested-By:ambari" -X POST http://{ambari-host}:{ambari-port}/api/v1/clusters/{clustername}/hosts/{hostname}/host_components/SL... 

Replace all the place holders in the curl call. Replace the {hostname} placeholder with the hostname where it is failing.

Thanks,

Aditya

[#3]

Thank you, @Aditya Sirna

I followed your instructions above and get the following error message.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 37, in <module>
    AfterInstallHook().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 31, in hook
    setup_stack_symlinks(self.stroutfile)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py", line 62, in setup_stack_symlinks
    stack_select.select(package, json_version)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 313, in select
    Execute(command, sudo=True)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/bin/hdp-select set oozie-client 2.6.4.0-91' returned 1. symlink target /usr/hdp/current/oozie-client for oozie already exists and it is not a symlink.

Step 1 and Step 3 went fine.But when I tried Step 2 and Step 4 I get the following responses and retry cluster installation failed. Should I actually delete those components first ?

{
  "status" : 409,
  "message" : "Attempted to create a host_component which already exists: [clusterName=bxp_singlenode_cluster, hostName=r00pvdn0c.bnymellon.net, componentName=OOZIE_CLIENT]"
}
{
  "status" : 409,
  "message" : "Attempted to create a host_component which already exists: [clusterName=bxp_singlenode_cluster, hostName=r00pvdn0c.bnymellon.net, componentName=SLIDER]"
}

[#4]

@Zaili Xu,

Step 2 and 4 should be done only if Step 1/3 gives 404. It was not clearly mentioned above. Can you please try deleting the folder /usr/hdp/current/oozie-client and run step 1 again. If you get the similar error for slider, do the same.

[#5]

I removed /usr/hdp/current/oozie-client and /usr/hdp/current/slider-client and re-run Step 1 and Step 3. Then I retried the cluster installation in Ambari console. Same error happened.

resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/bin/hdp-select set oozie-client 2.6.4.0-91' returned 1. symlink target /usr/hdp/current/oozie-client for oozie already exists and it is not a symlink.

And the /usr/hdp/current/oozie-client and /usr/hdp/current/slider-client came back. The ambari database in postgresql shows

select * from hostcomponentstate where service_name='OOZIE';
259	8	OOZIE_SERVER	UNKNOWN	INSTALL_FAILED	1	OOZIE	NONE	UNSECURED
285	8	OOZIE_CLIENT	UNKNOWN	INSTALL_FAILED	1	OOZIE	NONE	UNSECURED

[#6]

I figured out why the /usr/hdp/current/oozie-client/conf directory keeps coming back. The oozie_client.py tried to create the conf directory under /usr/hdp/current/oozie-client which is supposed to be a symbolic link but is missing. There for the actual directory is created /usr/hdp/current/oozie-client with a conf subdirectory. This in turn caused the error, "symlink target /usr/hdp/current/oozie-client for oozie already exists and it is not a symlink".

Now that I got the symbolic link created for oozie-client

/usr/hdp/current/oozie-client ->/usr/hdp/2.6.4.0-91/oozie

But oozie is missing under /usr/hdp/2.6.4.0-91. How do I make Ambari create it ? Will it help if I delete /var/lib/ambari-agent/cache ? @Aditya Sirna