Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Knox Gateway Start fail

Labels:
avatar
author-rank balajeeulaganat
Explorer

Created ‎04-05-2016 01:46 PM

Knox gateway failed to start after installation using ambari.

os : rhel 6.5

ambari : 2.1.2-377

knox : 0.6.0.2.3

java :jdk1.8.0_40

stderr: /var/lib/ambari-agent/data/errors-556.txt
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 267, in <module>
    KnoxGateway().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py", line 159, in start
    not_if=no_op_test
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 260, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/knox-server/bin/gateway.sh start' returned 1. Starting Gateway failed.
stdout: /var/lib/ambari-agent/data/output-556.txt
2016-04-05 18:41:17,752 - Group['spark'] {}
2016-04-05 18:41:17,754 - Group['hadoop'] {}
2016-04-05 18:41:17,754 - Group['users'] {}
2016-04-05 18:41:17,754 - Group['knox'] {}
2016-04-05 18:41:17,755 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,756 - User['storm'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,757 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,758 - User['oozie'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,758 - User['atlas'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,759 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,760 - User['falcon'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,761 - User['tez'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,762 - User['accumulo'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,763 - User['mahout'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,764 - User['spark'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,765 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-05 18:41:17,766 - User['flume'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,766 - User['kafka'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,767 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,769 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,770 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,770 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,771 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,772 - User['knox'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,773 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-05 18:41:17,774 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-04-05 18:41:17,776 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-04-05 18:41:17,788 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-04-05 18:41:17,788 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-04-05 18:41:17,789 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-04-05 18:41:17,790 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-04-05 18:41:17,795 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-04-05 18:41:17,796 - Group['hdfs'] {'ignore_failures': False}
2016-04-05 18:41:17,796 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2016-04-05 18:41:17,797 - Directory['/etc/hadoop'] {'mode': 0755}
2016-04-05 18:41:17,813 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-04-05 18:41:17,814 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-04-05 18:41:17,825 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-04-05 18:41:17,844 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-04-05 18:41:17,846 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-04-05 18:41:17,847 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-04-05 18:41:17,852 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-04-05 18:41:17,854 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-04-05 18:41:17,855 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-04-05 18:41:17,866 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-04-05 18:41:17,867 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-04-05 18:41:17,868 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-04-05 18:41:17,874 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-04-05 18:41:17,878 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-04-05 18:41:18,156 - HDP version to use is 2.3.4.0
2016-04-05 18:41:18,156 - Detected HDP with stack version 2.3.4.0-3485, will use knox_data_dir = /usr/hdp/2.3.4.0-3485/knox/data
2016-04-05 18:41:18,160 - Directory['/usr/hdp/current/knox-server/data/'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,162 - Directory['/var/log/knox'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,163 - Directory['/var/run/knox'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,163 - Directory['/usr/hdp/current/knox-server/conf'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,164 - Directory['/usr/hdp/current/knox-server/conf/topologies'] {'owner': 'knox', 'group': 'knox', 'recursive': True}
2016-04-05 18:41:18,164 - XmlConfig['gateway-site.xml'] {'owner': 'knox', 'group': 'knox', 'conf_dir': '/usr/hdp/current/knox-server/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-04-05 18:41:18,178 - Generating config: /usr/hdp/current/knox-server/conf/gateway-site.xml
2016-04-05 18:41:18,179 - File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] {'owner': 'knox', 'content': InlineTemplate(...), 'group': 'knox', 'mode': None, 'encoding': 'UTF-8'}
2016-04-05 18:41:18,185 - File['/usr/hdp/current/knox-server/conf/gateway-log4j.properties'] {'content': ..., 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-04-05 18:41:18,193 - File['/usr/hdp/current/knox-server/conf/topologies/default.xml'] {'content': InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
2016-04-05 18:41:18,194 - Execute[('chown', '-R', 'knox:knox', '/usr/hdp/current/knox-server/data/', '/var/log/knox', '/var/run/knox', '/usr/hdp/current/knox-server/conf', '/usr/hdp/current/knox-server/conf/topologies')] {'sudo': True}
2016-04-05 18:41:18,202 - Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_40'}, 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f /usr/hdp/current/knox-server/data/security/master'", 'user': 'knox'}
2016-04-05 18:41:18,301 - Skipping Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master [PROTECTED]'] due to not_if
2016-04-05 18:41:18,302 - Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-cert --hostname fssstrat.fss.india'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_40'}, 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f /usr/hdp/current/knox-server/data/security/keystores/gateway.jks'", 'user': 'knox'}
2016-04-05 18:41:18,377 - Skipping Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-cert --hostname fssstrat.fss.india'] due to not_if
2016-04-05 18:41:18,378 - File['/usr/hdp/current/knox-server/conf/ldap-log4j.properties'] {'content': ..., 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-04-05 18:41:18,379 - File['/usr/hdp/current/knox-server/conf/users.ldif'] {'content': ..., 'owner': 'knox', 'group': 'knox', 'mode': 0644}
2016-04-05 18:41:18,379 - Ranger admin not installed
2016-04-05 18:41:18,379 - Link['/usr/hdp/current/knox-server/pids'] {'to': '/var/run/knox'}
2016-04-05 18:41:18,380 - Execute['/usr/hdp/current/knox-server/bin/gateway.sh start'] {'environment': {'JAVA_HOME': '/usr/jdk64/jdk1.8.0_40'}, 'not_if': 'ls /var/run/knox/gateway.pid >/dev/null 2>&1 && ps -p `cat /var/run/knox/gateway.pid` >/dev/null 2>&1', 'user': 'knox'}
13,161 Views
11 REPLIES 11

avatar
author-rank hkropp
Super Collaborator

Created ‎04-05-2016 02:24 PM

Can you also provide what you find in /var/log/knox/gateway.log

8,662 Views

avatar
author-rank amiller
Guru

Created ‎04-05-2016 06:50 PM

You mention Knox 0.6.0 however the path shows 0.5.0. For Java it will also help to know whether you are using Oracle or OpenJDK. To address these questions, please also provide the output of:

hdp-select versions
hdp-select status knox-server
rpm -qa | grep knox
java -version
8,662 Views

avatar
author-rank kevin_minder
Guru

Created ‎04-05-2016 02:27 PM

and /var/log/knox/gateway.out if it isn't empty while you are collecting things.

8,662 Views
0 Kudos

avatar
author-rank balajeeulaganat
Explorer

Created ‎04-06-2016 04:29 AM

@Kevin Minder it is empty

8,662 Views
0 Kudos

avatar
author-rank balajeeulaganat
Explorer

Created ‎04-06-2016 04:22 AM

@Alex Miller Knox version ambari it shows as knox : 0.6.0.2.3 and java is openJDK 

hdp-select versions

2.3.4.0-3485

hdp-select status knox-server

knox-server - 2.3.4.0-3485

rpm -qa | grep knox

ranger_2_3_4_0_3485-knox-plugin-0.5.0.2.3.4.0-3485.el6.x86_64

knox_2_3_4_0_3485-0.6.0.2.3.4.0-3485.el6.noarch

8,662 Views

avatar
author-rank balajeeulaganat
Explorer

Created ‎04-06-2016 04:24 AM

@hkropp

/var/log/knox/gateway.log

2016-04-05 18:41:18,937 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(280)) - Loading configuration resource jar:file:/usr/hdp/2.3.4.0-3485/knox/bin/../lib/gateway-server-0.6.0.2.3.4.0-3485.jar!/conf/gateway-default.xml
2016-04-05 18:41:18,946 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigFile(268)) - Loading configuration file /usr/hdp/2.3.4.0-3485/knox/bin/../conf/gateway-site.xml
2016-04-05 18:41:18,964 INFO  hadoop.gateway (GatewayConfigImpl.java:initGatewayHomeDir(212)) - Using /usr/hdp/2.3.4.0-3485/knox/bin/.. as GATEWAY_HOME via system property.
2016-04-05 18:41:19,457 INFO  hadoop.gateway (JettySSLService.java:init(89)) - Credential store for the gateway instance found - no need to create one.
2016-04-05 18:41:19,458 INFO  hadoop.gateway (JettySSLService.java:init(106)) - Keystore for the gateway instance found - no need to create one.
2016-04-05 18:41:19,460 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(128)) - The Gateway SSL certificate is issued to hostname: fssstrat.fss.india.
2016-04-05 18:41:19,461 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(131)) - The Gateway SSL certificate is valid between: 4/6/16 3:03 AM and 4/6/17 3:03 AM.
2016-04-05 18:41:19,466 FATAL hadoop.gateway (GatewayServer.java:main(121)) - Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start.
2016-04-05 19:13:06,743 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(280)) - Loading configuration resource jar:file:/usr/hdp/2.3.4.0-3485/knox/bin/../lib/gateway-server-0.6.0.2.3.4.0-3485.jar!/conf/gateway-default.xml
2016-04-05 19:13:06,751 INFO  hadoop.gateway (GatewayConfigImpl.java:loadConfigFile(268)) - Loading configuration file /usr/hdp/2.3.4.0-3485/knox/bin/../conf/gateway-site.xml
2016-04-05 19:13:06,768 INFO  hadoop.gateway (GatewayConfigImpl.java:initGatewayHomeDir(212)) - Using /usr/hdp/2.3.4.0-3485/knox/bin/.. as GATEWAY_HOME via system property.
2016-04-05 19:13:07,257 INFO  hadoop.gateway (JettySSLService.java:init(89)) - Credential store for the gateway instance found - no need to create one.
2016-04-05 19:13:07,258 INFO  hadoop.gateway (JettySSLService.java:init(106)) - Keystore for the gateway instance found - no need to create one.
2016-04-05 19:13:07,260 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(128)) - The Gateway SSL certificate is issued to hostname: fssstrat.fss.india.
2016-04-05 19:13:07,261 INFO  hadoop.gateway (JettySSLService.java:logAndValidateCertificate(131)) - The Gateway SSL certificate is valid between: 4/6/16 3:03 AM and 4/6/17 3:03 AM.
2016-04-05 19:13:07,266 FATAL hadoop.gateway (GatewayServer.java:main(121)) - Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start.
8,662 Views

avatar
author-rank hkropp
Super Collaborator

Created ‎04-06-2016 09:51 AM

The Error message in /var/log/knox/gateway.log says that the certificate used by Knox will be valid starting in the future:

Failed to start gateway: org.apache.hadoop.gateway.services.ServiceLifecycleException: Gateway SSL Certificate is not yet valid. Server will not start.

-> "not yet valid"

Knox refuses to start, because using such a certificate will result in an SSL exception for almost any client.

You will need to check the certificate your are using for Knox. This is stored as gateway-identity in gateway.jks under /var/lib/knox/data*/keystore

Please refer to this:

http://knox.apache.org/books/knox-0-6-0/user-guide.html#Management+of+Security+Artifacts

What also should work is, if you simply remove the gateway-identity from the keystore, upon start Knox should create a self-signed certificate for you.

Could you share how the certificate was generated? Did you change it after the install? Are you using ntp?

8,662 Views

avatar
author-rank jose_agc
Contributor

Created ‎12-29-2016 09:17 PM

Renew Knox Gateway SSL certificate following the link:

http://www-01.ibm.com/support/docview.wss?uid=swg21987527

8,662 Views

avatar
author-rank helmi_khalifa
Contributor

Created ‎06-20-2019 09:11 AM

Hi


rename the file gateway.jks mv /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks.bck

when you start the know instance it will create a new certificate.


Best,

Helmi KHALIFA

8,662 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements