question Re: Not able to access WebHDFS via Knox in the Sandbox in Archives of Support Questions (Read Only)

Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Wed, 28 Sep 2016 01:44:05 GMT

Hello guys, I am using HDP 2.5 Sandbox and have started Knox as well as Demo LDAP service. Now when I am running:

curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'

I am getting the following message-

<html>

<head>

<title>Error 503 </title>

</head>

<body>

<h2>HTTP ERROR: 503</h2>

<p>Problem accessing /gateway/knox_sample/webhdfs/v1. Reason:

<pre> Service Unavailable</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

Has anyone experienced this before? Thanks in advance

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Thu, 29 Sep 2016 05:35:57 GMT

@mrizvi

A couple of things to check would be your knox_sample topology has the service mapping to webhdfs

<service> <role>WEBHDFS</role> <url>http://sandbox:50070/webhdfs</url> </service>

and the webhdfs server is listening on that port

netstat -nl | grep 50070

Take a look at your knox logs as well - they may have some extra info.

/usr/hdp/current/knox-server/logs/gateway.log

Cheers

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Thu, 29 Sep 2016 06:19:02 GMT

Thank you @Andrew Ryan for replying. Checked /usr/hdp/current/knox-server/conf/topologies/knox_sample.xml, it has this section:

<service>

<role>WEBHDFS</role>

<url>http://sandbox.hortonworks.com:50070/webhdfs</url>

</service>

Also, webhdfs server is listening to the port 50070.

Log file is not stating any error:

2016-09-28 18:53:39,285 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: user.name=knox

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: user.dir=/home/knox

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.runtime.name=OpenJDK Runtime Environment

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.runtime.version=1.7.0_111-mockbuild_2016_07_27_10_11-b00

2016-09-28 18:53:39,287 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre

2016-09-28 18:53:39,522 INFO hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(321)) - Loading configuration resource jar:file:/usr/hdp/2.5.0.0-1245/knox/bin/../lib/gateway-server-0.9.0.2.5.0.0-1245.jar!/conf/gateway-default.xml

2016-09-28 18:53:39,530 INFO hadoop.gateway (GatewayConfigImpl.java:loadConfigFile(309)) - Loading configuration file /usr/hdp/2.5.0.0-1245/knox/bin/../conf/gateway-site.xml

2016-09-28 18:53:39,614 INFO hadoop.gateway (GatewayConfigImpl.java:initGatewayHomeDir(253)) - Using /usr/hdp/2.5.0.0-1245/knox/bin/.. as GATEWAY_HOME via system property.

2016-09-28 18:53:40,111 INFO hadoop.gateway (JettySSLService.java:init(95)) - Credential store for the gateway instance found - no need to create one.

2016-09-28 18:53:40,151 INFO hadoop.gateway (JettySSLService.java:init(117)) - Keystore for the gateway instance found - no need to create one.

2016-09-28 18:53:40,156 INFO hadoop.gateway (JettySSLService.java:logAndValidateCertificate(146)) - The Gateway SSL certificate is issued to hostname: sandbox.hortonworks.com.

2016-09-28 18:53:40,157 INFO hadoop.gateway (JettySSLService.java:logAndValidateCertificate(149)) - The Gateway SSL certificate is valid between: 9/13/16 10:56 AM and 9/13/17 10:56 AM.

2016-09-28 18:53:40,454 INFO hadoop.gateway (GatewayServer.java:startGateway(279)) - Starting gateway...

2016-09-28 18:53:40,797 INFO hadoop.gateway (GatewayServer.java:start(379)) - Loading topologies from directory: /usr/hdp/2.5.0.0-1245/knox/bin/../conf/topologies

2016-09-28 18:53:40,950 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology knoxsso from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/knoxsso.topo.157239f6c28

2016-09-28 18:53:40,951 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology knoxsso

2016-09-28 18:53:40,974 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology knoxsso archive %2F

2016-09-28 18:53:41,015 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology knoxsso archive %2Fknoxauth

2016-09-28 18:53:41,100 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology admin from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/admin.topo.15723310670

2016-09-28 18:53:41,101 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology admin

2016-09-28 18:53:41,102 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology admin archive %2F

2016-09-28 18:53:41,103 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology default from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/default.topo.15723310670

2016-09-28 18:53:41,103 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology default

2016-09-28 18:53:41,104 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology default archive %2F

2016-09-28 18:53:41,105 INFO hadoop.gateway (GatewayServer.java:handleCreateDeployment(655)) - Loading topology knox_sample from /usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/knox_sample.topo.157233d52e0

2016-09-28 18:53:41,105 INFO hadoop.gateway (GatewayServer.java:internalActivateTopology(524)) - Activating topology knox_sample

2016-09-28 18:53:41,106 INFO hadoop.gateway (GatewayServer.java:internalActivateArchive(534)) - Activating topology knox_sample archive %2F

2016-09-28 18:53:41,289 INFO hadoop.gateway (GatewayServer.java:start(395)) - Monitoring topologies in directory: /usr/hdp/2.5.0.0-1245/knox/bin/../conf/topologies

2016-09-28 18:53:41,290 INFO hadoop.gateway (GatewayServer.java:startGateway(294)) - Started gateway on port 8,443.

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Thu, 29 Sep 2016 11:05:32 GMT

@mrizvi

No worries.

Can you contact the service without knox?

curl -i -v -k -u guest:guest-password 'http://sandbox.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS'

I have attached a topology file that currently works on my sandbox for you to compare.

Cheers

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Thu, 29 Sep 2016 11:21:37 GMT

Yes, I am able to contact webhdfs without Knox. Further, I am not able to access the sample file that you have attached.

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Thu, 29 Sep 2016 14:39:33 GMT

@mrizvi

Apologies, format issues I think. Try this one.knox-sample.xml

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Thu, 29 Sep 2016 23:46:33 GMT

Hello @Andrew Ryan , I just compared both of them and they are same. Also found out that I am not able to connect to other topologies like default, admin and knoxsso as well.

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Fri, 30 Sep 2016 06:42:33 GMT

@mrizvi

I would bump up the log levels for the gateway to DEBUG through Ambari as a next step.

In the knox advanced tab for gateway log4j change log4j.rootLogger=ERROR, drfa to log4j.rootLogger=DEBUG, drfa

Then submit a curl request check the gateway.log. There will be a lot of output so search for 'guest'.

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Fri, 30 Sep 2016 06:52:34 GMT

There was nothing for guest, but got something though it is just a warning:

WARN webapp.WebAppContext (WebAppContext.java:doStart(514)) - Failed startup of context o.e.j.w.WebAppContext@62811100{/gateway/knox_sample,null,null}{/usr/hdp/2.5.0.0-1245/knox/bin/../data/deployments/knox_sample.topo.157233d52e0/%2F}

java.lang.IllegalStateException: Failed to delete temp dir /var/lib/knox/data-2.5.0.0-1245/deployments/knox_sample.topo.157233d52e0/%2F/META-INF/temp

This warning is there for all 4 topologies

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Fri, 30 Sep 2016 08:48:24 GMT

@mrizvi

I don't get that warning so it may be significant. I'm not sure. If I had to hazard a guess as to why it's failing to delete it's either a path or file permissions issue. Can you post the whole log?

Also can you try a few tests with the /usr/hdp/current/knox-server.bin/knoxcli.sh utility? It might provide a bit more meaningful info.

./knoxcli.sh service-test --cluster knox_sample --u guest -p guest-password --hostname sandbox.hortonworks.com --port 8443 --cluster knox_sample

./knoxcli.sh user-auth-test --cluster knox_sample --u guest --p guest-password --hostname sandbox.hortonworks.com --port 8443 --cluster knox_sample

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Fri, 30 Sep 2016 08:56:42 GMT

Might be a good idea to check if you have any knox zombies running. They can hang on to files and prevent deletes.

ps -ef | grep -i knox

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Fri, 30 Sep 2016 11:23:38 GMT

@Andrew Ryan

Thank you for trying so hard with me. I tried both commands, first one did not get it through. Ends with the following message:

This command requires a running instance of Knox to be present on the same machine. It will execute a test to make sure all services are accessible through the gateway URLs. Errors are reported and suggestions to resolve any problems are returned. JSON formatted.

Second one showed the message that LDAP authentication got successful.

Also checked that there are no knox zombies running.

I have attached the gateway.log file.gatewaylog.txt

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Fri, 30 Sep 2016 15:03:08 GMT

@mrizvi

Quite alright. Maybe a long shot but I found this post with a similar issue to the one you are experiencing.

service unavailable

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Fri, 30 Sep 2016 15:36:47 GMT

It's a different error from the gateway logs in the attached log file for the link I posted. So it probably isn't your issue.

I'm confused now. This implies that knox isn't running. But the results for the user auth test say that it is. I'm running 2.4 so I will try to setup 2.5 over the weekend for myself.

Do you have ranger enabled with knox? Can you post the gateway.log error with debug enabled for when you make your curl request?

Cheers

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Fri, 30 Sep 2016 23:43:33 GMT

Hi @Andrew Ryan, Yes, ranger is enabled with knox. Please find the attached gateway.log file. I also went through gateway-audit.log file which shows this:

16/09/30 01:13:22 |||audit|||||redeploy|topology|knoxsso|unavailable|

16/09/30 01:13:23 |||audit|||||redeploy|topology|admin|unavailable|

16/09/30 01:13:23 |||audit|||||redeploy|topology|default|unavailable|

16/09/30 01:13:23 |||audit|||||redeploy|topology|knox_sample|unavailable|

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Wed, 05 Oct 2016 14:07:40 GMT

@mrizvi

I downloaded the 2.5 sandbox and got the same issue as you describe.

The problem seems to be the directory for previous deployments can't be deleted and this causes the service for the topologies to fail to start.

I eventually got mine working by moving all of the topology xml files out of /usr/hdp/current/knox-server/conf/topologies

and restarting knox. It automatically populates the default, knoxsso and admin files back into the folder. I was able to list the files using the default topology. I moved the knox_sample.xml back into the folder and did another restart. It failed to start due to the temp folder being unable to be deleted. So I catted the knox_sample.xml into another file knox_sample2.xml and restarted again. I was able to list the files through knox_sample2.

It is more of a work around than anything else. I don't know why the temp folder can't be deleted. I couldn't delete the folder manually and when I tried I got an invalid argument error

rmdir

/var/lib/knox/data-2.5.0.0-1245/deployments/knoxsso.topo.157239f6c28/%2Fknoxauth/META-INF/temp/jsp

rmdir: failed to remove `jsp/': Invalid argument

Re: Not able to access WebHDFS via Knox in the Sandbox

andrew_ryan1 — Wed, 05 Oct 2016 14:58:06 GMT

Knox_sample also works now

Re: Not able to access WebHDFS via Knox in the Sandbox

mrizvi — Wed, 05 Oct 2016 21:43:46 GMT

Thanks alot @Andrew Ryan, I moved all the files from the topologies folder, and then restarted Knox, it created default, admin and knoxsso xml files. Tested default and it worked fine. Then I moved back the old knox_sample.xml to the folder and restarted Knox, it worked. Still wondering what could be the issue