Created 09-27-2016 06:44 PM
Hello guys, I am using HDP 2.5 Sandbox and have started Knox as well as Demo LDAP service. Now when I am running:
curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
I am getting the following message-
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 </title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /gateway/knox_sample/webhdfs/v1. Reason:
<pre> Service Unavailable</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>
Has anyone experienced this before? Thanks in advance
Created 10-05-2016 07:07 AM
I downloaded the 2.5 sandbox and got the same issue as you describe.
The problem seems to be the directory for previous deployments can't be deleted and this causes the service for the topologies to fail to start.
I eventually got mine working by moving all of the topology xml files out of /usr/hdp/current/knox-server/conf/topologies
and restarting knox. It automatically populates the default, knoxsso and admin files back into the folder. I was able to list the files using the default topology. I moved the knox_sample.xml back into the folder and did another restart. It failed to start due to the temp folder being unable to be deleted. So I catted the knox_sample.xml into another file knox_sample2.xml and restarted again. I was able to list the files through knox_sample2.
It is more of a work around than anything else. I don't know why the temp folder can't be deleted. I couldn't delete the folder manually and when I tried I got an invalid argument error
rmdir
/var/lib/knox/data-2.5.0.0-1245/deployments/knoxsso.topo.157239f6c28/%2Fknoxauth/META-INF/temp/jsp
rmdir: failed to remove `jsp/': Invalid argument
Created 09-30-2016 01:56 AM
Might be a good idea to check if you have any knox zombies running. They can hang on to files and prevent deletes.
ps -ef | grep -i knox
Created 09-30-2016 04:23 AM
Thank you for trying so hard with me. I tried both commands, first one did not get it through. Ends with the following message:
This command requires a running instance of Knox to be present on the same machine. It will execute a test to make sure all services are accessible through the gateway URLs. Errors are reported and suggestions to resolve any problems are returned. JSON formatted.
Second one showed the message that LDAP authentication got successful.
Also checked that there are no knox zombies running.
I have attached the gateway.log file.gatewaylog.txt
Created 09-30-2016 08:03 AM
Quite alright. Maybe a long shot but I found this post with a similar issue to the one you are experiencing.
Created 09-30-2016 08:36 AM
It's a different error from the gateway logs in the attached log file for the link I posted. So it probably isn't your issue.
This command requires a running instance of Knox to be present on the same machine. It will execute a test to make sure all services are accessible through the gateway URLs. Errors are reported and suggestions to resolve any problems are returned. JSON formatted.
I'm confused now. This implies that knox isn't running. But the results for the user auth test say that it is. I'm running 2.4 so I will try to setup 2.5 over the weekend for myself.
Do you have ranger enabled with knox? Can you post the gateway.log error with debug enabled for when you make your curl request?
Cheers
Created 09-30-2016 04:43 PM
Hi @Andrew Ryan, Yes, ranger is enabled with knox. Please find the attached gateway.log file. I also went through gateway-audit.log file which shows this:
16/09/30 01:13:22 |||audit|||||redeploy|topology|knoxsso|unavailable|
16/09/30 01:13:23 |||audit|||||redeploy|topology|admin|unavailable|
16/09/30 01:13:23 |||audit|||||redeploy|topology|default|unavailable|
16/09/30 01:13:23 |||audit|||||redeploy|topology|knox_sample|unavailable|
Created 10-05-2016 07:07 AM
I downloaded the 2.5 sandbox and got the same issue as you describe.
The problem seems to be the directory for previous deployments can't be deleted and this causes the service for the topologies to fail to start.
I eventually got mine working by moving all of the topology xml files out of /usr/hdp/current/knox-server/conf/topologies
and restarting knox. It automatically populates the default, knoxsso and admin files back into the folder. I was able to list the files using the default topology. I moved the knox_sample.xml back into the folder and did another restart. It failed to start due to the temp folder being unable to be deleted. So I catted the knox_sample.xml into another file knox_sample2.xml and restarted again. I was able to list the files through knox_sample2.
It is more of a work around than anything else. I don't know why the temp folder can't be deleted. I couldn't delete the folder manually and when I tried I got an invalid argument error
rmdir
/var/lib/knox/data-2.5.0.0-1245/deployments/knoxsso.topo.157239f6c28/%2Fknoxauth/META-INF/temp/jsp
rmdir: failed to remove `jsp/': Invalid argument
Created 10-05-2016 07:58 AM
Knox_sample also works now
Created 10-05-2016 02:43 PM
Thanks alot @Andrew Ryan, I moved all the files from the topologies folder, and then restarted Knox, it created default, admin and knoxsso xml files. Tested default and it worked fine. Then I moved back the old knox_sample.xml to the folder and restarted Knox, it worked. Still wondering what could be the issue