Created 01-13-2018 08:37 PM
See my answer about fixing stack_root. "I tried upgrading the management pack after, but didn't help." It looks like you HAVE to do mpack first then Ambari upgrade going from Ambari 2.5.x to Ambari 2.6.x. This will automatically update the stack_root.
Created 01-13-2018 06:48 PM
The issue seems to be Ambari doesn't like the JSON coming back from Stack.get_stack_root(). The format looks like it changed between Ambari 2.5.1 and Ambari 2.5.2 with AMBARI-21430. The commit looks like this: f33a250c0
This is what the new Ambari 2.6 cluster has for stack_root.
"stack_root" : "{\"HDF\":\"/usr/hdf\"}",
This is what our upgraded Ambari 2.6 cluster has for stack_root.
"stack_root" : "/usr/hdf",
This can easily be updated with configs.py since this is just cluster-env -> stack_root. An example of doing this:
/var/lib/ambari-server/resources/scripts/configs.py -l $(hostname -f) -t 8081 -s https -n CLUSTER_NAME -u USERNAME -p PASSWORD -a set -c cluster-env -k stack_root -v '{"HDF":"/usr/hdf"}'
PS: It also looks like this requires that you have the 3.0.2 mpack installed (even if you have 3.0.1 HDF installed) since that is compatible with Ambari 2.6.x (see matrix here). Without the 3.0.2 mpack, we were getting different errors (like the ones in the original question).
To install 3.0.2 mpack (make sure purge flag is set):
sudo ambari-server install-mpack --mpack=https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.2.0-76.tar.gz --purge --verbose --force
Created 01-16-2018 01:35 PM
Hello @Kevin Risden,
I have not been able to solve it with configs.py, (probably) because I already tried without success other solutions (that may have interfered).
Anyway your hint about inverting the order of the upgrades has been helpful, the correct order is: first upgrade the mpack then upgrade Ambari. Thanks.
On this matter the Hortonworks HDF Upgrade documentation is confusing, it seems to suggest the wrong order (and leaves the idea that you can follow both orders).
Raffaele
Created 01-19-2018 02:01 AM
@Raffaele S - We found a few more things that were broken which we were able to fix. I am working on writing up some more steps for this. We were able to get everything to work. I agree with you that the documentation isn't clear that mpack must go before Ambari upgrade. I'll update this question with more info hopefully soon.
Created 02-08-2018 02:34 AM
@Raffaele S Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html
Created 02-01-2018 08:55 PM
Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?
Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?
Created 02-02-2018 09:48 AM
Did you backup the database, resource folders etc. as described in the upgrade guide?
If you did, restore all the backups, downgrade ambari to the previous version and follow the proper order (as I described above).
Created 02-02-2018 12:18 PM
Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.
Created 02-08-2018 02:35 AM
Created 02-02-2018 12:17 PM
Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.