Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Knox Gateway Start fail

avatar

Knox gateway failed to start after installation using ambari.

os : rhel 6.5

ambari : 2.1.2-377

knox : 0.6.0.2.3

java :jdk1.8.0_40

stderr: /var/lib/ambari-agent/data/errors-556.txt
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 267, in <module>
    KnoxGateway().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 159, in start
    not_if=no_op_test
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 260, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/knox-server/bin/gateway.sh start' returned 1. Starting Gateway failed.
stdout: /var/lib/ambari-agent/data/output-556.txt
2016-04-05 18:41:17,752 - Group['spark'] {}
2016-04-05 18:41:17,754 - Group['hadoop'] {}
2016-04-05 18:41:17,754 - Group['users'] {}
2016-04-05 18:41:17,754 - Group['knox'] {}
2016-04-05 18:41:17,755 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,756 - User['storm'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,757 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,758 - User['oozie'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,758 - User['atlas'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,759 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,760 - User['falcon'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,761 - User['tez'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,762 - User['accumulo'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,763 - User['mahout'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,764 - User['spark'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,765 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,766 - User['flume'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,766 - User['kafka'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,767 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,769 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,770 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,770 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,771 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,772 - User['knox'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,773 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,774 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-04-05 18:41:17,776 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-04-05 18:41:17,788 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-04-05 18:41:17,788 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-04-05 18:41:17,789 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-04-05 18:41:17,790 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-04-05 18:41:17,795 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-04-05 18:41:17,796 - Group['hdfs'] {'ignore_failures': False}
2016-04-05 18:41:17,796 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2016-04-05 18:41:17,797 - Directory['/etc/hadoop'] {'mode': 0755}
2016-04-05 18:41:17,813 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-04-05 18:41:17,814 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-04-05 18:41:17,825 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-04-05 18:41:17,844 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-04-05 18:41:17,846 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-04-05 18:41:17,847 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-04-05 18:41:17,852 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-04-05 18:41:17,854 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-04-05 18:41:17,855 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-04-05 18:41:17,866 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-04-05 18:41:17,867 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-04-05 18:41:17,868 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-04-05 18:41:17,874 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-04-05 18:41:17,878 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-04-05 18:41:18,156 - HDP version to use is 2.3.4.0
2016-04-05 18:41:18,156 - Detected HDP with stack version 2.3.4.0-3485, will use knox_data_dir = /usr/hdp/2.3.4.0-3485/knox/data
2016-04-05 18:41:18,160 - Directory['/usr/hdp/current/knox-server/data/'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,162 - Directory['/var/log/knox'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,163 - Directory['/var/run/knox'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,163 - Directory['/usr/hdp/current/knox-server/conf'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,164 - Directory['/usr/hdp/current/knox-server/conf/topologies'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,164 - XmlConfig['gateway-site.xml'] {'owner': 'knox', 'group': 'knox', 'conf_dir': '/usr/hdp/current/knox-server/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-04-05 18:41:18,178 - Generating config: /usr/hdp/current/knox-server/conf/gateway-site.xml
2016-04-05 18:41:18,179 - File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] {'owner': 'knox', 'content': InlineTemplate(...), 'group': 'knox', 'mode': None, 'encoding': 'UTF-8'}
2016-04-05 18:41:18,185 - File['/usr/hdp/current/knox-server/conf/gateway-log4j.properties'] {'content': ..., 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-04-05 18:41:18,193 - File['/usr/hdp/current/knox-server/conf/topologies/default.xml'] {'content': InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
2016-04-05 18:41:18,194 - Execute[('chown', '-R', 'knox:knox', '/usr/hdp/current/knox-server/data/', '/var/log/knox', '/var/run/knox', '/usr/hdp/current/knox-server/conf', '/usr/hdp/current/knox-server/conf/topologies')] {'sudo': True}
2016-04-05 18:41:18,202 - Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_40'}, 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f /usr/hdp/current/knox-server/data/security/master'", 'user': 'knox'}
2016-04-05 18:41:18,301 - Skipping Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]'] due to not_if
2016-04-05 18:41:18,302 - Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-cert --hostname fssstrat.fss.india'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_40'}, 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f /usr/hdp/current/knox-server/data/security/keystores/gateway.jks'", 'user': 'knox'}
2016-04-05 18:41:18,377 - Skipping Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-cert --hostname fssstrat.fss.india'] due to not_if
2016-04-05 18:41:18,378 - File['/usr/hdp/current/knox-server/conf/ldap-log4j.properties'] {'content': ..., 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-04-05 18:41:18,379 - File['/usr/hdp/current/knox-server/conf/users.ldif'] {'content': ..., 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-04-05 18:41:18,379 - Ranger admin not installed
2016-04-05 18:41:18,379 - Link['/usr/hdp/current/knox-server/pids'] {'to': '/var/run/knox'}
2016-04-05 18:41:18,380 - Execute['/usr/hdp/current/knox-server/bin/gateway.sh start'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_40'}, 'not_if': 'ls /var/run/knox/gateway.pid >/dev/null 2>&1 && ps -p `cat /var/run/knox/gateway.pid` >/dev/null 2>&1', 'user': 'knox'}
11 REPLIES 11

avatar
Super Collaborator

Can you also provide what you find in /var/log/knox/gateway.log

avatar

You mention Knox 0.6.0 however the path shows 0.5.0. For Java it will also help to know whether you are using Oracle or OpenJDK. To address these questions, please also provide the output of:

hdp-select versions
hdp-select status knox-server
rpm -qa | grep knox
java -version

avatar

and /var/log/knox/gateway.out if it isn't empty while you are collecting things.

avatar

@Kevin Minder it is empty

avatar

@Alex Miller Knox version ambari it shows as knox : 0.6.0.2.3 and java is openJDK

hdp-select versions

2.3.4.0-3485

hdp-select status knox-server

knox-server - 2.3.4.0-3485

rpm -qa | grep knox

ranger_2_3_4_0_3485-knox-plugin-0.5.0.2.3.4.0-3485.el6.x86_64

knox_2_3_4_0_3485-0.6.0.2.3.4.0-3485.el6.noarch

avatar

@hkropp

/var/log/knox/gateway.log

2016-04-05 18:41:18,937 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(280)) - Loading configuration resource jar:file:/usr/hdp/2.3.4.0-3485/knox/bin/../lib/gateway-server-0.6.0.2.3.4.0-3485.jar!/conf/gateway-default.xml
2016-04-05 18:41:18,946 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigFile(268)) - Loading configuration file /usr/hdp/2.3.4.0-3485/knox/bin/../conf/gateway-site.xml
2016-04-05 18:41:18,964 INFO  hadoop.gateway (GatewayConfigImpl.java:initGatewayHomeDir(212)) - Using /usr/hdp/2.3.4.0-3485/knox/bin/.. as GATEWAY_HOME via system property.
2016-04-05 18:41:19,457 INFO  hadoop.gateway (JettySSLService.java:init(89)) - Credential store for the gateway instance found - no need to create one.
2016-04-05 18:41:19,458 INFO  hadoop.gateway (JettySSLService.java:init(106)) - Keystore for the gateway instance found - no need to create one.
2016-04-05 18:41:19,460 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(128)) - The Gateway SSL certificate is issued to hostname: fssstrat.fss.india.
2016-04-05 18:41:19,461 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(131)) - The Gateway SSL certificate is valid between: 4/6/16 3:03 AM and 4/6/17 3:03 AM.
2016-04-05 18:41:19,466 FATAL hadoop.gateway (GatewayServer.java:main(121)) - Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start.
2016-04-05 19:13:06,743 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(280)) - Loading configuration resource jar:file:/usr/hdp/2.3.4.0-3485/knox/bin/../lib/gateway-server-0.6.0.2.3.4.0-3485.jar!/conf/gateway-default.xml
2016-04-05 19:13:06,751 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigFile(268)) - Loading configuration file /usr/hdp/2.3.4.0-3485/knox/bin/../conf/gateway-site.xml
2016-04-05 19:13:06,768 INFO  hadoop.gateway (GatewayConfigImpl.java:initGatewayHomeDir(212)) - Using /usr/hdp/2.3.4.0-3485/knox/bin/.. as GATEWAY_HOME via system property.
2016-04-05 19:13:07,257 INFO  hadoop.gateway (JettySSLService.java:init(89)) - Credential store for the gateway instance found - no need to create one.
2016-04-05 19:13:07,258 INFO  hadoop.gateway (JettySSLService.java:init(106)) - Keystore for the gateway instance found - no need to create one.
2016-04-05 19:13:07,260 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(128)) - The Gateway SSL certificate is issued to hostname: fssstrat.fss.india.
2016-04-05 19:13:07,261 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(131)) - The Gateway SSL certificate is valid between: 4/6/16 3:03 AM and 4/6/17 3:03 AM.
2016-04-05 19:13:07,266 FATAL hadoop.gateway (GatewayServer.java:main(121)) - Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start.



avatar
Super Collaborator

The Error message in /var/log/knox/gateway.log says that the certificate used by Knox will be valid starting in the future:

Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start.

-> "not yet valid"

Knox refuses to start, because using such a certificate will result in an SSL exception for almost any client.

You will need to check the certificate your are using for Knox. This is stored as gateway-identity in gateway.jks under /var/lib/knox/data*/keystore

Please refer to this:

http://knox.apache.org/books/knox-0-6-0/user-guide.html#Management+of+Security+Artifacts

What also should work is, if you simply remove the gateway-identity from the keystore, upon start Knox should create a self-signed certificate for you.

Could you share how the certificate was generated? Did you change it after the install? Are you using ntp?

avatar
Contributor

Renew Knox Gateway SSL certificate following the link:

http://www-01.ibm.com/support/docview.wss?uid=swg21987527

avatar
Contributor

Hi


rename the file gateway.jks mv /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks.bck

when you start the know instance it will create a new certificate.


Best,

Helmi KHALIFA