Support Questions
Find answers, ask questions, and share your expertise

Re: Knox Gateway Start fail

New Contributor

Hello,

I'm having the exact same issue when installing the latest version of HDP2.3 a couple of hours ago.

During the ambari-server setup installation process, I chose to install the Oracle JDK 1.8 (jdk1.8.0_40).

Will try the installation again tomorrow from scratch with Oracle JDK 1.7.

Cedric

Re: Knox Gateway Start fail

New Contributor

Installing with Oracle JDK 1.7 instead of JDK 1.8 did not make a difference.

Result of Knox gateway start:

2016-01-15 15:37:00,354 - Directory['/var/lib/knox/data'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-01-15 15:37:00,357 - Directory['/var/log/knox'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-01-15 15:37:00,357 - Directory['/var/run/knox'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-01-15 15:37:00,358 - Directory['/usr/hdp/current/knox-server/conf'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-01-15 15:37:00,359 - Directory['/usr/hdp/current/knox-server/conf/topologies'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-01-15 15:37:00,359 - XmlConfig['gateway-site.xml'] {'owner': 'knox', 'group': 'knox', 'conf_dir': '/usr/hdp/current/knox-server/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-01-15 15:37:00,386 - Generating config: /usr/hdp/current/knox-server/conf/gateway-site.xml
2016-01-15 15:37:00,387 - File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] {'owner': 'knox', 'content': InlineTemplate(...), 'group': 'knox', 'mode': None, 'encoding': 'UTF-8'}
2016-01-15 15:37:00,399 - Writing File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] because contents don't match
2016-01-15 15:37:00,400 - File['/usr/hdp/current/knox-server/conf/gateway-log4j.properties'] {'content': '...', 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-01-15 15:37:00,411 - File['/usr/hdp/current/knox-server/conf/topologies/default.xml'] {'content': InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
2016-01-15 15:37:00,412 - Execute['('chown', '-R', u'knox:knox', '/var/lib/knox/data', '/var/log/knox', u'/var/run/knox', '/usr/hdp/current/knox-server/conf', '/usr/hdp/current/knox-server/conf/topologies')'] {'sudo': True}
2016-01-15 15:37:00,440 - Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]'] {'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.7.0_67'}, 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f /var/lib/knox/data/security/master'", 'user': 'knox'}
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 264, in <module>
    KnoxGateway().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 146, in start
    self.configure(env)
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 63, in configure
    knox()
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox.py", line 125, in knox
    not_if=master_secret_exist,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 258, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]' returned 1. Master secret is already present on disk. Please be aware that overwriting it will require updating other security artifacts.  Use --force to overwrite the existing master secret.
ERROR: Invalid Command
Unrecognized option:create-master
A fatal exception has occurred. Program will exit.

Re: Knox Gateway Start fail

New Contributor

Looking at the command that causes the error:

2016-01-15 15:37:00,440 - Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]'] {'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.7.0_67'}, 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f /var/lib/knox/data/security/master'", 'user': 'knox'}

Originates to /var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox.py

cmd = format('{knox_client_bin} create-master --master {knox_master_secret!p}')
master_secret_exist = as_user(format('test -f {knox_master_secret_path}'), params.knox_user)
Execute(cmd,user=params.knox_user,environment={'JAVA_HOME': params.java_home},not_if=master_secret_exist,)

{knox_master_secret_path} resolves to /var/lib/knox/data/security/master (as defined in /var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py). The problem is that the Knox master file does not exist on this location. The directory /var/lib/knox/data does exist, but the content is empty.

Instead, the master key is located here: /usr/hdp/current/knox-server/data/security/master

In the file /var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py, I also see something with removing/setting symbolic links:

# Used to setup symlink, needed to update the knox managed symlink, in case of custom locations
if os.path.islink(params.knox_managed_pid_symlink) and os.path.realpath(params.knox_managed_pid_symlink) != params.knox_pid_dir:
os.unlink(params.knox_managed_pid_symlink)
os.symlink(params.knox_pid_dir, params.knox_managed_pid_symlink)

Perhaps something goes wrong with the symbolic links? (when you install HDP2.3 successfully, but try to restart all services immediately after the installation?)

----

In any case, the following modification resolved the issue for me.. I'm not sure if it covers everything (e.g. what will happen if you change the Knox master key via the Ambari web interface??), but I don't have any more time to be stuck on this issue 🙂

Open /var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py

Change:

knox_master_secret_path = '/var/lib/knox/data/security/master' 
knox_cert_store_path = '/var/lib/knox/data/security/keystores/gateway.jks'

to:

knox_master_secret_path = '/usr/hdp/current/knox-server/data/security/master' 
knox_cert_store_path = '/usr/hdp/current/knox-server/data/security/keystores/gateway.jks'

Looking forward to a response from a Hortonworks employee..

View solution in original post

Re: Knox Gateway Start fail

As of HDP 2.2.4, I believe that Knox changed its data folder to be a symlink to a versioned folder, so

/usr/hdp/current/knox-server/data -> /var/lib/knox/data/${version} (or something like this)

This means that Knox in Ambari should always follow the symlink to get the right path instead of trying to access /var/lib/knox/data directly.

I opened AMBARI-14726 to track a fix.

Re: Knox Gateway Start fail

Actually, that issue is fixed in Ambari 2.1.2 by https://issues.apache.org/jira/browse/AMBARI-12979

Re: Knox Gateway Start fail

New Contributor

OK. I didn't realize that I didn't install the latest version.

Since my goal is to obtain the HDPCA certificate, I used the reference links from http://hortonworks.com/training/class/hdp-certified-administrator-hdpca-exam/. Unfortunately they are not up to date.

I guess I will also be trying the upgrade process then.. 😛

Re: Knox Gateway Start fail

Explorer

Thank you Cedric. The fix you provided also worked for me immediately.

Open /var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py Change:

  1. knox_master_secret_path ='/var/lib/knox/data/security/master'
  2. knox_cert_store_path ='/var/lib/knox/data/security/keystores/gateway.jks'

to:

  1. knox_master_secret_path ='/usr/hdp/current/knox-server/data/security/master'
  2. knox_cert_store_path ='/usr/hdp/current/knox-server/data/security/keystores/gateway.jks'

Re: Knox Gateway Start fail

Explorer

As Cedric Colpaert points out in his answer, for me the best, you must change the params_linux.py file on both agent and server directories:

/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts

/var/lib/ambari-server/resources/common-services/KNOX/0.5.0.2.2/package/scripts

Re: Knox Gateway Start fail

New Contributor

I just ran into the same issue, also with the latest version of Ambari & Knox, on CentOS 7 with Java 1.8. FWIW, I found that if I rm /usr/hdp/current/knox-server/data/security/master, the service starts OK from within Ambari.

Re: Knox Gateway Start fail

If you see this error, make sure to upgrade Ambari to version 2.1.2 or higher,

https://github.com/apache/ambari/blob/release-2.1.2-rc3/ambari-server/src/main/resources/common-serv...