HDF3 installation problem

[#1]

Hi everyone,

when I arrive step 9 (install, start and test) i have this error : I try to install HDF3 (NIFI/ZOOKEEPER) using Ambari on one node (REDHAT7) , I go through all the steps but when

stderr: /var/lib/ambari-agent/data/errors-5.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 35, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 26, in hook
    import params	




  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/params.py", line 101, in <module>
    hadoop_home = stack_select.get_hadoop_dir("home", force_latest_on_upgrade=True)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/stack_select.py", line 210, in get_hadoop_dir
    stack_root = Script.get_stack_root()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 575, in get_stack_root
    stack_root = json.loads(stack_root_json)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-5.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-5.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']

stdout: /var/lib/ambari-agent/data/output-5.txt

2017-10-05 10:26:08,957 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-5.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-5.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2017-10-05 10:26:09,363 - Cannot find stack features for the stack named HDF

Command failed after 1 tries

I can not find a solution for this, any tips?

Best Regards

[#2 · Accepted solution]

Hi @badr bakkou,

can you please check the ambari version(both server and agents) which you have chosen, it must be of 2.5.1, at the same time please ensure that you have the appropriate management pack as well (compatible version mentioned below.)

please find the compatible matrix here

Hope that helps !!

[#3]

Hi @bkosaraju,

i use ambari version 2.5.2.

I think the problem comes from the incompatibility of the version, i switch to 2.5.1 and i try again i keep you infromed.

thanks.

Best Regards.

[#4]

it's work with he correct version. thanks man

[#5]

Hi,

I have this same error but it occurred during the upgrade process of Ambari from 2.5.1.0-159 to 2.6.0.0-267. The Ambari upgrade itself went through fine and the server and agents started. But during the post-upgrade steps those services (Infra, Log Search, etc) all fail with this error.

I've confirmed the versions of my server and all agents are the same but is there perhaps something else I might've missed?

Regards,

Geouffrey

[#6]

Hello @Geouffrey Erasmus,

I have exactly the same problem, any update?

Thanks.

[#7]

Hey @Raffaele S,

Unfortunately not. In the end I reverted to backups to HDF 3.0.1 and performed an upgrade to 3.0.1.1 instead, which is using Ambari 2.5.1. This version is fine for now as I was trying to resolve issues with Schema Registry but it is a bit of a pain as I would've preferred upgrading to the latest version while I was going through the upgrade pain.

Hope you have better luck

Regards,

Geouffrey

[#8]

Thanks @Geouffrey Erasmus.

Luckily this happened on a test cluster, we ended up reinstalling from scratch.

I will write here if the same happens on our development cluster.

Raffaele

[#9]

@Raffaele S, That is a bummer, I feel your pain. I had a similar scenario a few months ago trying to upgrade to HDF 3 and ended up reinstalling and configuring an entire cluster.

Hope it goes better with the next upgrade.

[#10]

Hello,

just happened again on our development cluster (RHEL 7.3). In this case reinstalling the cluster is not feasible.

All my agents are version: ambari-agent-2.6.0.0-267.x86_64. My ambari-server: ambari-server-2.6.0.0-267.x86_64

A bit of context, I reached this point of the ambari upgrade procedure: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_ambari-upgrade/content/upgrade_ambari.ht...

After accessing the console I noticed Zookeeper, Storm and metrics were down (this is not correct, the services are running).

Restarting zookeeper led to this error:

stderr: /var/lib/ambari-agent/data/errors-1202.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py", line 40, in <module>
    BeforeStartHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py", line 28, in hook
    import params
  File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/params.py", line 76, in <module>
    hadoop_libexec_dir = stack_select.get_hadoop_dir("libexec")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/stack_select.py", line 365, in get_hadoop_dir
    stack_root = Script.get_stack_root()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 659, in get_stack_root
    stack_root = json.loads(stack_root_json)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-1202.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-1202.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']

stdout: /var/lib/ambari-agent/data/output-1202.txt

2018-01-08 15:40:58,562 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.1.1-5 -> 3.0.1.1-5
2018-01-08 15:40:58,564 - Cannot find stack features for the stack named HDF
2018-01-08 15:40:58,566 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-1202.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-1202.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2018-01-08 15:40:58,792 - Cannot find stack features for the stack named HDF

Command failed after 1 tries

I tried upgrading the management pack after, but didn't help.

In this case, should I open a JIRA?

[#11]

See my answer about fixing stack_root. "I tried upgrading the management pack after, but didn't help." It looks like you HAVE to do mpack first then Ambari upgrade going from Ambari 2.5.x to Ambari 2.6.x. This will automatically update the stack_root.

[#12]

The issue seems to be Ambari doesn't like the JSON coming back from Stack.get_stack_root(). The format looks like it changed between Ambari 2.5.1 and Ambari 2.5.2 with AMBARI-21430. The commit looks like this: f33a250c0

This is what the new Ambari 2.6 cluster has for stack_root.

"stack_root" : "{\"HDF\":\"/usr/hdf\"}", 

This is what our upgraded Ambari 2.6 cluster has for stack_root.

"stack_root" : "/usr/hdf", 

This can easily be updated with configs.py since this is just cluster-env -> stack_root. An example of doing this:

/var/lib/ambari-server/resources/scripts/configs.py -l $(hostname -f) -t 8081 -s https -n CLUSTER_NAME -u USERNAME -p PASSWORD -a set -c cluster-env -k stack_root -v '{"HDF":"/usr/hdf"}' 

PS: It also looks like this requires that you have the 3.0.2 mpack installed (even if you have 3.0.1 HDF installed) since that is compatible with Ambari 2.6.x (see matrix here). Without the 3.0.2 mpack, we were getting different errors (like the ones in the original question).

To install 3.0.2 mpack (make sure purge flag is set):

sudo ambari-server install-mpack --mpack=https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.2.0-76.tar.gz --purge --verbose --force

[#13]

Hello @Kevin Risden,

I have not been able to solve it with configs.py, (probably) because I already tried without success other solutions (that may have interfered).

Anyway your hint about inverting the order of the upgrades has been helpful, the correct order is: first upgrade the mpack then upgrade Ambari. Thanks.

On this matter the Hortonworks HDF Upgrade documentation is confusing, it seems to suggest the wrong order (and leaves the idea that you can follow both orders).

Raffaele

[#14]

@Raffaele S - We found a few more things that were broken which we were able to fix. I am working on writing up some more steps for this. We were able to get everything to work. I agree with you that the documentation isn't clear that mpack must go before Ambari upgrade. I'll update this question with more info hopefully soon.

[#15]

@Raffaele S Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html

[#16]

Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?

,

Hi @Kevin Risden, I also ran upgrade before updating mpack. Can you please share the information how to fix it?

[#17]

Did you backup the database, resource folders etc. as described in the upgrade guide?

If you did, restore all the backups, downgrade ambari to the previous version and follow the proper order (as I described above).

[#18]

Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.

[#19]

@Devendra Singh Here are all the details in a blog: https://risdenk.github.io/2018/02/07/hdf-ambari-mpack-upgrades.html

[#20]

Unfortunately I haven't done backup prior to upgrade... any suggestion to fix this issue will be really appreciated.

[#21]

HI ALL,

I Have SAME ISSUE TO INSTALL NIFI USING HDF3.1 .AMBARI-SERVER AND AMBARI-AGENT SAME VERSION 2.6.1.0.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py", line 35, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py", line 29, in hook
    setup_users()
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py", line 51, in setup_users
    fetch_nonlocal_groups = params.fetch_nonlocal_groups,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/accounts.py", line 84, in action_create
    shell.checked_call(command, sudo=True)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'usermod -G nifi -g hadoop nifi' returned 6. usermod: user 'nifi' does not exist in /etc/passwd
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-2103.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-2103.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''