HDF3 installation problem

[#1]

Hi everyone,

when I arrive step 9 (install, start and test) i have this error : I try to install HDF3 (NIFI/ZOOKEEPER) using Ambari on one node (REDHAT7) , I go through all the steps but when

stderr: /var/lib/ambari-agent/data/errors-5.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 35, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 26, in hook
    import params	




  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/params.py", line 101, in <module>
    hadoop_home = stack_select.get_hadoop_dir("home", force_latest_on_upgrade=True)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/stack_select.py", line 210, in get_hadoop_dir
    stack_root = Script.get_stack_root()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 575, in get_stack_root
    stack_root = json.loads(stack_root_json)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-5.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-5.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']

stdout: /var/lib/ambari-agent/data/output-5.txt

2017-10-05 10:26:08,957 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-5.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-5.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2017-10-05 10:26:09,363 - Cannot find stack features for the stack named HDF

Command failed after 1 tries

I can not find a solution for this, any tips?

Best Regards

[#2]

See my answer about fixing stack_root. "I tried upgrading the management pack after, but didn't help." It looks like you HAVE to do mpack first then Ambari upgrade going from Ambari 2.5.x to Ambari 2.6.x. This will automatically update the stack_root.

[#3]

The issue seems to be Ambari doesn't like the JSON coming back from Stack.get_stack_root(). The format looks like it changed between Ambari 2.5.1 and Ambari 2.5.2 with AMBARI-21430. The commit looks like this: f33a250c0

This is what the new Ambari 2.6 cluster has for stack_root.

"stack_root" : "{\"HDF\":\"/usr/hdf\"}", 

This is what our upgraded Ambari 2.6 cluster has for stack_root.

"stack_root" : "/usr/hdf", 

This can easily be updated with configs.py since this is just cluster-env -> stack_root. An example of doing this:

/var/lib/ambari-server/resources/scripts/configs.py -l $(hostname -f) -t 8081 -s https -n CLUSTER_NAME -u USERNAME -p PASSWORD -a set -c cluster-env -k stack_root -v '{"HDF":"/usr/hdf"}' 

PS: It also looks like this requires that you have the 3.0.2 mpack installed (even if you have 3.0.1 HDF installed) since that is compatible with Ambari 2.6.x (see matrix here). Without the 3.0.2 mpack, we were getting different errors (like the ones in the original question).

To install 3.0.2 mpack (make sure purge flag is set):

sudo ambari-server install-mpack --mpack=https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.2.0-76.tar.gz --purge --verbose --force

[#4]

Hello @Kevin Risden,

I have not been able to solve it with configs.py, (probably) because I already tried without success other solutions (that may have interfered).

Anyway your hint about inverting the order of the upgrades has been helpful, the correct order is: first upgrade the mpack then upgrade Ambari. Thanks.

On this matter the Hortonworks HDF Upgrade documentation is confusing, it seems to suggest the wrong order (and leaves the idea that you can follow both orders).

Raffaele

[#5]

@Raffaele S - We found a few more things that were broken which we were able to fix. I am working on writing up some more steps for this. We were able to get everything to work. I agree with you that the documentation isn't clear that mpack must go before Ambari upgrade. I'll update this question with more info hopefully soon.

[#6]

@Raffaele S Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html

[#7]

Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?

,

Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?

[#8]

Did you backup the database, resource folders etc. as described in the upgrade guide?

If you did, restore all the backups, downgrade ambari to the previous version and follow the proper order (as I described above).

[#9]

Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.

[#10]

@Devendra Singh Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html

[#11]

Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.