Support Questions

mrizvi · ‎09-27-2016

Hello guys, I am using HDP 2.5 Sandbox and have started Knox as well as Demo LDAP service. Now when I am running:

curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'

I am getting the following message-

<html>

<head>

<title>Error 503 </title>

</head>

<body>

<h2>HTTP ERROR: 503</h2>

<p>Problem accessing /gateway/knox_sample/webhdfs/v1. Reason:

<pre> Service Unavailable</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

Has anyone experienced this before? Thanks in advance

andrew_ryan1 · ‎10-05-2016

@mrizvi

I downloaded the 2.5 sandbox and got the same issue as you describe.

The problem seems to be the directory for previous deployments can't be deleted and this causes the service for the topologies to fail to start.

I eventually got mine working by moving all of the topology xml files out of /usr/hdp/current/knox-server/conf/topologies

and restarting knox. It automatically populates the default, knoxsso and admin files back into the folder. I was able to list the files using the default topology. I moved the knox_sample.xml back into the folder and did another restart. It failed to start due to the temp folder being unable to be deleted. So I catted the knox_sample.xml into another file knox_sample2.xml and restarted again. I was able to list the files through knox_sample2.

It is more of a work around than anything else. I don't know why the temp folder can't be deleted. I couldn't delete the folder manually and when I tried I got an invalid argument error

rmdir

/var/lib/knox/data-2.5.0.0-1245/deployments/knoxsso.topo.157239f6c28/%2Fknoxauth/META-INF/temp/jsp

rmdir: failed to remove `jsp/': Invalid argument

View solution in original post

andrew_ryan1 · ‎09-28-2016

@mrizvi

A couple of things to check would be your knox_sample topology has the service mapping to webhdfs

<service> <role>WEBHDFS</role> <url>http://sandbox:50070/webhdfs</url> </service>

and the webhdfs server is listening on that port

netstat -nl | grep 50070

Take a look at your knox logs as well - they may have some extra info.

/usr/hdp/current/knox-server/logs/gateway.log

Cheers

mrizvi · ‎09-28-2016

Thank you @Andrew Ryan for replying. Checked /usr/hdp/current/knox-server/conf/topologies/knox_sample.xml, it has this section:

<service>

<role>WEBHDFS</role>

<url>http://sandbox.hortonworks.com:50070/webhdfs</url>

</service>

Also, webhdfs server is listening to the port 50070.

Log file is not stating any error:

2016-09-28 18:53:39,285 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: user.name=knox

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: user.dir=/home/knox

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.runtime.name=OpenJDK Runtime Environment

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.runtime.version=1.7.0_111-mockbuild_2016_07_27_10_11-b00

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre

2016-09-28 18:53:39,522 INFO hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(321)) - Loading configuration resource jar:file:/usr/hdp/2.5.0.0-1245/knox/bin/../lib/gateway-server-0.9.0.2.5.0.0-1245.jar!/conf/gateway-default.xml

2016-09-28 18:53:39,530 INFO hadoop.gateway (GatewayConfigImpl.java:loadConfigFile(309)) - Loading configuration file /usr/hdp/2.5.0.0-1245/knox/bin/../conf/gateway-site.xml

2016-09-28 18:53:39,614 INFO hadoop.gateway (GatewayConfigImpl.java:initGatewayHomeDir(253)) - Using /usr/hdp/2.5.0.0-1245/knox/bin/.. as GATEWAY_HOME via system property.

2016-09-28 18:53:40,111 INFO hadoop.gateway (JettySSLService.java:init(95)) - Credential store for the gateway instance found - no need to create one.

2016-09-28 18:53:40,151 INFO hadoop.gateway (JettySSLService.java:init(117)) - Keystore for the gateway instance found - no need to create one.

2016-09-28 18:53:40,156 INFO hadoop.gateway (JettySSLService.java:logAndValidateCertificate(146)) - The Gateway SSL certificate is issued to hostname: sandbox.hortonworks.com.

2016-09-28 18:53:40,157 INFO hadoop.gateway (JettySSLService.java:logAndValidateCertificate(149)) - The Gateway SSL certificate is valid between: 9/13/16 10:56 AM and 9/13/17 10:56 AM.

2016-09-28 18:53:40,454 INFO hadoop.gateway (GatewayServer.java:startGateway(279)) - Starting gateway...

2016-09-28 18:53:40,797 INFO hadoop.gateway (GatewayServer.java:start(379)) - Loading topologies from directory: /usr/hdp/2.5.0.0-1245/knox/bin/../conf/topologies

2016-09-28 18:53:40,950 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology knoxsso from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/knoxsso.topo.157239f6c28

2016-09-28 18:53:40,951 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology knoxsso

2016-09-28 18:53:40,974 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology knoxsso archive %2F

2016-09-28 18:53:41,015 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology knoxsso archive %2Fknoxauth

2016-09-28 18:53:41,100 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology admin from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/admin.topo.15723310670

2016-09-28 18:53:41,101 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology admin

2016-09-28 18:53:41,102 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology admin archive %2F

2016-09-28 18:53:41,103 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology default from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/default.topo.15723310670

2016-09-28 18:53:41,103 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology default

2016-09-28 18:53:41,104 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology default archive %2F

2016-09-28 18:53:41,105 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology knox_sample from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/knox_sample.topo.157233d52e0

2016-09-28 18:53:41,105 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology knox_sample

2016-09-28 18:53:41,106 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology knox_sample archive %2F

2016-09-28 18:53:41,289 INFO hadoop.gateway (GatewayServer.java:start(395)) - Monitoring topologies in directory: /usr/hdp/2.5.0.0-1245/knox/bin/../conf/topologies

2016-09-28 18:53:41,290 INFO hadoop.gateway (GatewayServer.java:startGateway(294)) - Started gateway on port 8,443.

andrew_ryan1 · ‎09-29-2016

@mrizvi

No worries.

Can you contact the service without knox?

curl -i -v -k -u guest:guest-password 'http://sandbox.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS'

I have attached a topology file that currently works on my sandbox for you to compare.

Cheers

mrizvi · ‎09-29-2016

Yes, I am able to contact webhdfs without Knox. Further, I am not able to access the sample file that you have attached.

andrew_ryan1 · ‎09-29-2016

@mrizvi

Apologies, format issues I think. Try this one.knox-sample.xml

mrizvi · ‎09-29-2016

Hello @Andrew Ryan , I just compared both of them and they are same. Also found out that I am not able to connect to other topologies like default, admin and knoxsso as well.

andrew_ryan1 · ‎09-29-2016

@mrizvi

I would bump up the log levels for the gateway to DEBUG through Ambari as a next step.

In the knox advanced tab for gateway log4j change log4j.rootLogger=ERROR, drfa to log4j.rootLogger=DEBUG, drfa

Then submit a curl request check the gateway.log. There will be a lot of output so search for 'guest'.

mrizvi · ‎09-29-2016

There was nothing for guest, but got something though it is just a warning:

WARN webapp.WebAppContext (WebAppContext.java:doStart(514)) - Failed startup of context o.e.j.w.WebAppContext@62811100{/gateway/knox_sample,null,null}{/usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/knox_sample.topo.157233d52e0/%2F}

java.lang.IllegalStateException: Failed to delete temp dir /var/lib/knox/data-2.5.0.0-1245/deployments/knox_sample.topo.157233d52e0/%2F/META-INF/temp

This warning is there for all 4 topologies

andrew_ryan1 · ‎09-30-2016

@mrizvi

I don't get that warning so it may be significant. I'm not sure. If I had to hazard a guess as to why it's failing to delete it's either a path or file permissions issue. Can you post the whole log?

Also can you try a few tests with the /usr/hdp/current/knox-server.bin/knoxcli.sh utility? It might provide a bit more meaningful info.

./knoxcli.sh service-test --cluster knox_sample --u guest -p guest-password --hostname sandbox.hortonworks.com --port 8443 --cluster knox_sample

./knoxcli.sh user-auth-test --cluster knox_sample --u guest --p guest-password --hostname sandbox.hortonworks.com --port 8443 --cluster knox_sample

Cloudera Community

Support Questions

Not able to access WebHDFS via Knox in the Sandbox