Last step of Ambari HDP installation fails for ZooKeeper client, on one of five nodes, due to "looped symbolic links" error

[#1]

@Rohit Sureka @Predrag Minovic

I'm trying to install a 5 node hadoop cluster with HDP 2.3.4.7 on Ubuntu 14.04 using Ambari 2.2.0.0. During the 10th step of the Ambari "Cluster Install Wizard" the installation of the ZooKeeper client fails on one of the five nodes with the following stderr:

"resource_management.core.exceptions.Fail: Applying Directory['/usr/hdp/current/zookeeper-client/conf'] failed, looped symbolic links found while resolving /usr/hdp/current/zookeeper-client/conf"

During the earlier steps of the "Cluster Install Wizard" and prior to this error, I removed the packages "zlibc" and "xclip" from two hosts, one of them now failing the ZooKeeper installation, in oder to comply with warnings telling me to do so. But "apt-get remove zlibc" and "apt-get remove xclip" also removed the Ambari agents on these hosts. Therefore the installation process didn't finish the first time because the heartbeat of these to hosts got lost during the installation. So I reinstalled the Ambari agents on these hosts but ever since I get the error mentioned above.

I hope someone can give me a hint or a solution. Thanks a lot.

The full stderr reads like this:

Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py", line 81, in <module> ZookeeperClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py", line 61, in install self.configure(env) File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py", line 37, in configure zookeeper(type='client') File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py", line 46, in zookeeper group=params.user_group File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 144, in action_create raise Fail("Applying %s failed, looped symbolic links found while resolving %s" % (self.resource, path)) resource_management.core.exceptions.Fail: Applying Directory['/usr/hdp/current/zookeeper-client/conf'] failed, looped symbolic links found while resolving /usr/hdp/current/zookeeper-client/conf

[#2 · Accepted solution]

@Felix Karabanov

I recently experience something similar during an upgrade. I am not sure what the root cause within the software but here is what Ambari is expecting and work around that worked for me -

Correct setup -

The configuration directories are expected to be setup this way -

Starting with /etc/zookeeper/conf

[root@sandbox zookeeper]# ll /etc/zookeeper/conf
lrwxrwxrwx 1 root root 38 2015-10-27 14:31 /etc/zookeeper/conf -> /usr/hdp/current/zookeeper-client/conf
[root@sandbox zookeeper]# ll /usr/hdp/current/zookeeper-client/conf
lrwxrwxrwx 1 root root 29 2015-10-27 14:31 /usr/hdp/current/zookeeper-client/conf -> /etc/zookeeper/2.3.2.0-2950/0

In your case, the link /usr/hdp/current/zookeeper-client/conf probably points back to /etc/zookeeper/conf, which causes the issue.

[#3]

Fix is to delete that link manually and recreate it correctly.

rm /usr/hdp/current/zookeeper-client/conf
ln -s /etc/zookeeper/2.3.2.0-2950/0 /usr/hdp/current/zookeeper-client/conf

[#4]

Thank you @bsaini it worked great.

[#5]

(+1) Hi Felix, also check that /usr/hdp/current/zookeeper-client is pointing to /usr/hdp/2.3.2.0-2950/zookeeper, and that directory /etc/zookeeper/2.3.3.0-2950/0 exists, if not create it. ZK config files like zoo.cfg and zookeeper-env.sh will be stored there.

[#6]

Was there an earlier install of HDP either successful or unsuccessful? Its important to cleanup failed installs or clean up existing installs for reinstalls. After host registration, ambari gives a good estimate of issues that can effect your install.

You can take a look at https://community.hortonworks.com/questions/1110/how-to-completely-remove-uninstall-ambari-and-hdp.h... though this is geared towards centos.

For this specific issue, you can unlink /usr/hdp/current/zookeeper-client/ and retry. This should take care of it.