Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

[#1]

Hi,

I have an issue at installation time of datanode (Ambari 2.1.2, HDP 2.2.4) , it fails because at step "Datanode start" with the error:

""

  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 87, in action_create
    raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] failed, parent directory /usr/hdp/current/hadoop-client/conf doesn't exist

""

Obviously the cause of this issue already happens at step "Datanode install", because the last log message is:

""

...

2016-01-12 19:44:17,233 - Skipping XmlConfig['core-site.xml'] due to only_if
2016-01-12 19:44:17,233 - Can only link configs for HDP-2.3 and higher.

""

Hmmm, strange, HDP2.2 should be supported by recent versions of Ambari...

Is this a known issue? any workarounds, hints what to do ?

Thanks, Gerd

[#2]

@Gerd Koenig you need to run hdp-select tool with the full version of the HDP you're running on the other servers. Check if you have /usr/hdp/2.x directory that matches your current version on other nodes, once that's confirmed, refer to the docs to execute hdp-select statement associated with your version, then you restart all services on the node and run service checks.

[#3]

just to add, most likely adding the node did not complete as hdp-select statement executed last to symlink /usr/hdp/2.x directory to /usr/hdp/current for client and server libs.

[#4]

Hi @Artem Ervits,

thanks for this input. I tried it, but the link named 'current' hasn't been created, seems like the hdp-select command didn't do anything?!?!

$ ls -al /usr/hdp
total 12
drwxr-xr-x  3 root root 4096 Jan 12 20:28 .
drwxr-xr-x 16 root root 4096 Jan 12 20:28 ..
drwxr-xr-x 13 root root 4096 Jan 13 08:56 2.2.4.2-2
$ /usr/bin/hdp-select set all 2.2.4.2-2
$ ls -al /usr/hdp
total 12
drwxr-xr-x  3 root root 4096 Jan 12 20:28 .
drwxr-xr-x 16 root root 4096 Jan 12 20:28 ..
drwxr-xr-x 13 root root 4096 Jan 13 08:56 2.2.4.2-2

[#5]

does /usr/hdp/2.x exist @Gerd Koenig? Do you have enough space on /usr?

[#6 · Accepted solution]

Hi @Artem Ervits, @Neeraj Sabharwal

just to give you an update on this:

- yes, there was/is enough space in /usr and the dir /usr/hdp/2.2.5.2-2 has been created

- due to time pressure I decided to start again from scratch with ambari 2.0.1 (the combination which I used for earlier clusters as well), right now it looks good, also that particular host is up as a worker node

- to cleanup all the nodes I used the hostcleanup python script and some additional directory deletions (due to non-standard directories, e.g. for logs)

I will do the Ambari upgrade later on.

Thanks for all you efforts