- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
REST API via Knox with Ranger permission get 307 Temporary Redirect message
- Labels:
-
Apache Knox
-
Apache Ranger
Created 10-11-2017 07:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All, to fix knox rest api 404 not found error, i modify WEBHDFS url from {{webhdfs_service_urls}} to http://My_NameNode:50070/webhdfs, it's worked.
then i using Ranger to control HDFS access permission, and configure Ranger HDFS and Knox plug-in...
i run curl command again, will display 307 Temporary Redirect!
i research about 307 message says maybe have name node HA and need add other name node url to Knox WEBHDFS setting,
but this is not match my ambari system because i have only one name node (no HA),
if any setting missing in Ranger or Knox?
Created 10-11-2017 08:04 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Sen Ke,
Hope you are trying to do some WEBHDFS operations. This is common for WEBHDFS.
Ex; If you want to upload a file using WEBHDFS , you have to make 2 curl calls. The response code of 1st curl call will be 307 redirect. The second call should be made to the Location Header obtained from the 1st curl call.
Sample Curl Call to Create and Write to a File
- Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE [&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>] [&permission=<OCTAL>][&buffersize=<INT>]"
The request is redirected to a datanode where the file data is to be written:HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE. Content-Length: 0
- Step 2: Submit another HTTP PUT request using the URL in the Location header with the file data to be written.
curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."
The client receives a 201 Created response with zero content length and the WebHDFS URI of the file in the Location header:HTTP/1.1 201 Created Location: webhdfs://<HOST>:<PORT>/<PATH> Content-Length: 0
The above URL is for normal WEBHDFS. To access it through Knox, you can use the knox url instead of HDFS.
You can read more about WEBHDFS here and Knox WEBHDFS here
Thanks,
Aditya
Created 10-11-2017 08:04 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Sen Ke,
Hope you are trying to do some WEBHDFS operations. This is common for WEBHDFS.
Ex; If you want to upload a file using WEBHDFS , you have to make 2 curl calls. The response code of 1st curl call will be 307 redirect. The second call should be made to the Location Header obtained from the 1st curl call.
Sample Curl Call to Create and Write to a File
- Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE [&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>] [&permission=<OCTAL>][&buffersize=<INT>]"
The request is redirected to a datanode where the file data is to be written:HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE. Content-Length: 0
- Step 2: Submit another HTTP PUT request using the URL in the Location header with the file data to be written.
curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."
The client receives a 201 Created response with zero content length and the WebHDFS URI of the file in the Location header:HTTP/1.1 201 Created Location: webhdfs://<HOST>:<PORT>/<PATH> Content-Length: 0
The above URL is for normal WEBHDFS. To access it through Knox, you can use the knox url instead of HDFS.
You can read more about WEBHDFS here and Knox WEBHDFS here
Thanks,
Aditya
Created 10-11-2017 08:38 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks your reply, but i tried as below (try to put a test file )
curl -i -k -u user1:Hadoop -X PUT 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/test?op=CREATE'
get 307 Temporary Redirect, then type
curl -i -k -u user1:Hadoop -X PUT -T /test 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/test?op=CREATE'
is it right? i get 307 again
Created 10-11-2017 08:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looks like your 1st call and second call are for same url. Hit the 1st url and capture the response
Sample response
- HTTP/1.1307 TEMPORARY_REDIRECT
- Location: http://<xxx>:<yyy>/webhdfs/v1/<PATH>?op=CREATE.
- Content-Length:0
Now make the 2nd curl call to the url obtained from the response headers of 1st call ( Location Header highlighted in bold ) (ie.. http://<xxx>:<yyy>/webhdfs/v1/<PATH>?op=CREATE...)
Thanks,
Aditya
Created 10-11-2017 10:10 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Adding sample example here without knox. It would be similar with knox as well.
[root@xx user]# curl -i -X PUT "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?user.name=livy&op=CREATE" HTTP/1.1 307 TEMPORARY_REDIRECT Cache-Control: no-cache Expires: Tue, 26 Sep 2017 17:33:17 GMT Date: Tue, 26 Sep 2017 17:33:17 GMT Pragma: no-cache Expires: Tue, 26 Sep 2017 17:33:17 GMT Date: Tue, 26 Sep 2017 17:33:17 GMT Pragma: no-cache X-FRAME-OPTIONS: SAMEORIGIN Set-Cookie: hadoop.auth="u=livy&p=livy&t=simple&e=1506483197716&s=dRvADKPG0lrenLje4fmEEdgChFw="; Path=/; HttpOnly Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&cre... Content-Type: application/octet-stream Content-Length: 0 Server: Jetty(6.1.26.hwx) ### second curl call with to Location obtained above [root@xxx user]# curl -i -T /tmp/a.txt "http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false"
Created 10-12-2017 08:10 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sen Ke, Was this issue resolved.
Created 10-12-2017 08:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i not sure, i copied Location url to another curl command but not success , i am checking if my command wrong or other error,
i will feedback my status after test.
Created 10-13-2017 02:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i can access webhdfs using curl now, but have a probelm
when curl via knox i only can use admin (-u admin:admin-password) and can't use other account or will reply HTTP/1.1 401 Unauthorized,
i has configured ranger and grand permission to other account (user1),
i can create/delete file with user1 in ambari file view via knox (user1 should have permission)
mycommand:
curl -i -k -u user1:Hadoop -X PUT 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/senfile1?op=CREATE'
folder permission:
drwxr-xr-x - user1 hdfs 0 2017-10-05 11:08 /user1
do you know what the problem?
Created 10-13-2017 04:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Glad that it worked for you. Can you please accept the answer and start a new thread for this so that the thread will not get deviated from the main topic. Just tag me in the new thread along with the gateway.log file. I guess the user1 is not added under Advanced users-ldif in Knox advanced config.
Thanks,
Aditya