question Re: HDF3 installation problem in Support Questions

HDF3 installation problem

badri0055 — Thu, 05 Oct 2017 16:07:02 GMT

Hi everyone,

when I arrive step 9 (install, start and test) i have this error : I try to install HDF3 (NIFI/ZOOKEEPER) using Ambari on one node (REDHAT7) , I go through all the steps but when

stderr: /var/lib/ambari-agent/data/errors-5.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 35, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 26, in hook
    import params	




  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/params.py", line 101, in <module>
    hadoop_home = stack_select.get_hadoop_dir("home", force_latest_on_upgrade=True)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/stack_select.py", line 210, in get_hadoop_dir
    stack_root = Script.get_stack_root()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 575, in get_stack_root
    stack_root = json.loads(stack_root_json)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-5.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-5.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']

stdout: /var/lib/ambari-agent/data/output-5.txt

2017-10-05 10:26:08,957 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-5.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-5.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2017-10-05 10:26:09,363 - Cannot find stack features for the stack named HDF

Command failed after 1 tries

I can not find a solution for this, any tips?

Best Regards

Re: HDF3 installation problem

bkosaraju — Thu, 05 Oct 2017 16:18:39 GMT

Hi @badr bakkou,

can you please check the ambari version(both server and agents) which you have chosen, it must be of 2.5.1, at the same time please ensure that you have the appropriate management pack as well (compatible version mentioned below.)

please find the compatible matrix here

Hope that helps !!

Re: HDF3 installation problem

badri0055 — Thu, 05 Oct 2017 17:09:16 GMT

Hi @bkosaraju,

i use ambari version 2.5.2.

I think the problem comes from the incompatibility of the version, i switch to 2.5.1 and i try again i keep you infromed.

thanks.

Best Regards.

Re: HDF3 installation problem

badri0055 — Fri, 06 Oct 2017 20:34:57 GMT

it's work with he correct version. thanks man

Re: HDF3 installation problem

Geouff — Fri, 08 Dec 2017 17:14:24 GMT

Hi,

I have this same error but it occurred during the upgrade process of Ambari from 2.5.1.0-159 to 2.6.0.0-267. The Ambari upgrade itself went through fine and the server and agents started. But during the post-upgrade steps those services (Infra, Log Search, etc) all fail with this error.

I've confirmed the versions of my server and all agents are the same but is there perhaps something else I might've missed?

Regards,

Geouffrey

Re: HDF3 installation problem

rsg — Wed, 20 Dec 2017 01:33:32 GMT

Hello @Geouffrey Erasmus,

I have exactly the same problem, any update?

Thanks.

Re: HDF3 installation problem

Geouff — Wed, 20 Dec 2017 22:24:54 GMT

Hey @Raffaele S,

Unfortunately not. In the end I reverted to backups to HDF 3.0.1 and performed an upgrade to 3.0.1.1 instead, which is using Ambari 2.5.1. This version is fine for now as I was trying to resolve issues with Schema Registry but it is a bit of a pain as I would've preferred upgrading to the latest version while I was going through the upgrade pain.

Hope you have better luck

Regards,

Geouffrey

Re: HDF3 installation problem

rsg — Wed, 03 Jan 2018 15:57:09 GMT

Thanks @Geouffrey Erasmus.

Luckily this happened on a test cluster, we ended up reinstalling from scratch.

I will write here if the same happens on our development cluster.

Raffaele

Re: HDF3 installation problem

Geouff — Wed, 03 Jan 2018 17:35:42 GMT

@Raffaele S, That is a bummer, I feel your pain. I had a similar scenario a few months ago trying to upgrade to HDF 3 and ended up reinstalling and configuring an entire cluster.

Hope it goes better with the next upgrade.

Re: HDF3 installation problem

rsg — Mon, 08 Jan 2018 22:45:38 GMT

Hello,

just happened again on our development cluster (RHEL 7.3). In this case reinstalling the cluster is not feasible.

All my agents are version: ambari-agent-2.6.0.0-267.x86_64. My ambari-server: ambari-server-2.6.0.0-267.x86_64

A bit of context, I reached this point of the ambari upgrade procedure: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_ambari-upgrade/content/upgrade_ambari.html

After accessing the console I noticed Zookeeper, Storm and metrics were down (this is not correct, the services are running).

Restarting zookeeper led to this error:

stderr: /var/lib/ambari-agent/data/errors-1202.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py", line 40, in <module>
    BeforeStartHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py", line 28, in hook
    import params
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/params.py", line 76, in <module>
    hadoop_libexec_dir = stack_select.get_hadoop_dir("libexec")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/stack_select.py", line 365, in get_hadoop_dir
    stack_root = Script.get_stack_root()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 659, in get_stack_root
    stack_root = json.loads(stack_root_json)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-1202.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-1202.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']

stdout: /var/lib/ambari-agent/data/output-1202.txt

2018-01-08 15:40:58,562 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.1.1-5 -> 3.0.1.1-5
2018-01-08 15:40:58,564 - Cannot find stack features for the stack named HDF
2018-01-08 15:40:58,566 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-1202.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-1202.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2018-01-08 15:40:58,792 - Cannot find stack features for the stack named HDF

Command failed after 1 tries

I tried upgrading the management pack after, but didn't help.

In this case, should I open a JIRA?

Re: HDF3 installation problem

risdenk — Sun, 14 Jan 2018 02:48:26 GMT

The issue seems to be Ambari doesn't like the JSON coming back from Stack.get_stack_root(). The format looks like it changed between Ambari 2.5.1 and Ambari 2.5.2 with AMBARI-21430. The commit looks like this: f33a250c0

This is what the new Ambari 2.6 cluster has for stack_root.

"stack_root" : "{\"HDF\":\"/usr/hdf\"}",

This is what our upgraded Ambari 2.6 cluster has for stack_root.

"stack_root" : "/usr/hdf",

This can easily be updated with configs.py since this is just cluster-env -> stack_root. An example of doing this:

/var/lib/ambari-server/resources/scripts/configs.py -l $(hostname -f) -t 8081 -s https -n CLUSTER_NAME -u USERNAME -p PASSWORD -a set -c cluster-env -k stack_root -v '{"HDF":"/usr/hdf"}'

PS: It also looks like this requires that you have the 3.0.2 mpack installed (even if you have 3.0.1 HDF installed) since that is compatible with Ambari 2.6.x (see matrix here). Without the 3.0.2 mpack, we were getting different errors (like the ones in the original question).

To install 3.0.2 mpack (make sure purge flag is set):

sudo ambari-server install-mpack --mpack=https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.2.0-76.tar.gz --purge --verbose --force

Re: HDF3 installation problem

risdenk — Sun, 14 Jan 2018 04:37:35 GMT

See my answer about fixing stack_root. "I tried upgrading the management pack after, but didn't help." It looks like you HAVE to do mpack first then Ambari upgrade going from Ambari 2.5.x to Ambari 2.6.x. This will automatically update the stack_root.

Re: HDF3 installation problem

rsg — Tue, 16 Jan 2018 21:35:41 GMT

Hello @Kevin Risden,

I have not been able to solve it with configs.py, (probably) because I already tried without success other solutions (that may have interfered).

Anyway your hint about inverting the order of the upgrades has been helpful, the correct order is: first upgrade the mpack then upgrade Ambari. Thanks.

On this matter the Hortonworks HDF Upgrade documentation is confusing, it seems to suggest the wrong order (and leaves the idea that you can follow both orders).

Raffaele

Re: HDF3 installation problem

risdenk — Fri, 19 Jan 2018 10:01:18 GMT

@Raffaele S - We found a few more things that were broken which we were able to fix. I am working on writing up some more steps for this. We were able to get everything to work. I agree with you that the documentation isn't clear that mpack must go before Ambari upgrade. I'll update this question with more info hopefully soon.

Re: HDF3 installation problem

dsingh — Fri, 02 Feb 2018 04:55:29 GMT

Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?

Re: HDF3 installation problem

rsg — Fri, 02 Feb 2018 17:48:37 GMT

Did you backup the database, resource folders etc. as described in the upgrade guide?

If you did, restore all the backups, downgrade ambari to the previous version and follow the proper order (as I described above).

Re: HDF3 installation problem

dsingh — Fri, 02 Feb 2018 20:17:16 GMT

Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.

Re: HDF3 installation problem

dsingh — Fri, 02 Feb 2018 20:18:13 GMT

Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.

Re: HDF3 installation problem

risdenk — Thu, 08 Feb 2018 10:34:44 GMT

@Raffaele S Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html

Re: HDF3 installation problem

risdenk — Thu, 08 Feb 2018 10:35:17 GMT

@Devendra Singh Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html