question Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing in Archives of Support Questions (Read Only)

Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 02:53:55 GMT

Hi,

I have an issue at installation time of datanode (Ambari 2.1.2, HDP 2.2.4) , it fails because at step "Datanode start" with the error:

  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 87, in action_create
    raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] failed, parent directory /usr/hdp/current/hadoop-client/conf doesn't exist

Obviously the cause of this issue already happens at step "Datanode install", because the last log message is:

...

2016-01-12 19:44:17,233 - Skipping XmlConfig['core-site.xml'] due to only_if
2016-01-12 19:44:17,233 - Can only link configs for HDP-2.3 and higher.

Hmmm, strange, HDP2.2 should be supported by recent versions of Ambari...

Is this a known issue? any workarounds, hints what to do ?

Thanks, Gerd

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

nsabharwal — Wed, 13 Jan 2016 02:55:56 GMT

@Gerd Koenig

I would use compatiable ambari version with HDP and then upgrade ambari to save time.

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 03:10:20 GMT

Hi @Neeraj Sabharwal,

thanks for getting back that quickly.

Referring to page: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_determine_stack_compatibility.html

I am assuming that Ambari 2.1.2 is supporting HDP 2.2.4 ?!?!

Which version should I use instead ?

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

nsabharwal — Wed, 13 Jan 2016 03:12:38 GMT

@Gerd Koenig You are right on the compatibility. Is it new install? or cleaning up existing install then installing new version ?

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

nsabharwal — Wed, 13 Jan 2016 03:14:05 GMT

@Gerd Koenig

whats the output of following?

ls -l /usr/hdp

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

afernandez — Wed, 13 Jan 2016 03:26:00 GMT

Hi Gerd, can you check the following,

rpm -qa | grep hadoop  # make sure it is installed and that /etc/hadoop/conf exists

ls -la /usr/hdp/current/hadoop-client  # should be a symlink to /etc/hadoop/conf, if not, create it

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 03:38:20 GMT

@Neeraj Sabharwal

ok, tried doing it again after removing any hadoop_* packages on that node from a previous run, but again:

...

2016-01-12 20:29:26,813 - Skipping XmlConfig['core-site.xml'] due to only_if
2016-01-12 20:29:26,814 - Can only link configs for HDP-2.3 and higher.

and then:

$ ls -al /usr/hdp
total 12
drwxr-xr-x  3 root root 4096 Jan 12 20:28 .
drwxr-xr-x 16 root root 4096 Jan 12 20:28 ..
drwxr-xr-x 13 root root 4096 Jan 12 20:29 2.2.4.2-2

hence, the link 'current' is missing...thereby the datanode fails at starting up

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 03:40:02 GMT

Maybe I will start completely from scratch with a fresh installation.

Which Ambari version would you recommend for setting up a fresh HDP 2.2.4 ? Yes, unfortunately I have to use that HDP version, an upgrade is planned for later this year 😉

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

nsabharwal — Wed, 13 Jan 2016 03:40:20 GMT

@Gerd Koenig Let's follow the correct approach to cleanup

https://community.hortonworks.com/content/idea/138/cleanup-scripts-to-remove-old-hdp-bits.html

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

aervits — Wed, 13 Jan 2016 13:40:49 GMT

@Gerd Koenig you need to run hdp-select tool with the full version of the HDP you're running on the other servers. Check if you have /usr/hdp/2.x directory that matches your current version on other nodes, once that's confirmed, refer to the docs to execute hdp-select statement associated with your version, then you restart all services on the node and run service checks.

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

aervits — Wed, 13 Jan 2016 13:42:33 GMT

just to add, most likely adding the node did not complete as hdp-select statement executed last to symlink /usr/hdp/2.x directory to /usr/hdp/current for client and server libs.

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 16:01:17 GMT

Hi @Alejandro Fernandez,

thanks for your answer, but it is not exactly how it should look like 😉 (maybe you are referring to an older version, before adding the version number into the installation directory tree).

Currently, the linking should be:

/usr/hdp/current -> /usr/hdp/2.2.4.2-2

and

/usr/hdp/2.2.4.2-2/hadoop/conf -> /etc/hadoop/conf

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 16:07:17 GMT

Hi @Artem Ervits,

thanks for this input. I tried it, but the link named 'current' hasn't been created, seems like the hdp-select command didn't do anything?!?!

$ ls -al /usr/hdp
total 12
drwxr-xr-x  3 root root 4096 Jan 12 20:28 .
drwxr-xr-x 16 root root 4096 Jan 12 20:28 ..
drwxr-xr-x 13 root root 4096 Jan 13 08:56 2.2.4.2-2
$ /usr/bin/hdp-select set all 2.2.4.2-2
$ ls -al /usr/hdp
total 12
drwxr-xr-x  3 root root 4096 Jan 12 20:28 .
drwxr-xr-x 16 root root 4096 Jan 12 20:28 ..
drwxr-xr-x 13 root root 4096 Jan 13 08:56 2.2.4.2-2

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

aervits — Wed, 13 Jan 2016 19:16:36 GMT

does /usr/hdp/2.x exist @Gerd Koenig? Do you have enough space on /usr?

Re: Ambari 2.1.2 : cannot start newly added datanode because /usr/hdp/current link is missing

geko — Wed, 13 Jan 2016 21:55:08 GMT

Hi @Artem Ervits, @Neeraj Sabharwal

just to give you an update on this:

- yes, there was/is enough space in /usr and the dir /usr/hdp/2.2.5.2-2 has been created

- due to time pressure I decided to start again from scratch with ambari 2.0.1 (the combination which I used for earlier clusters as well), right now it looks good, also that particular host is up as a worker node

- to cleanup all the nodes I used the hostcleanup python script and some additional directory deletions (due to non-standard directories, e.g. for logs)

I will do the Ambari upgrade later on.

Thanks for all you efforts