question Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message in Support Questions

REST API via Knox with Ranger permission get 307 Temporary Redirect message

sen_ke — Wed, 11 Oct 2017 14:42:07 GMT

Hi All, to fix knox rest api 404 not found error, i modify WEBHDFS url from {{webhdfs_service_urls}} to http://My_NameNode:50070/webhdfs, it's worked.

then i using Ranger to control HDFS access permission, and configure Ranger HDFS and Knox plug-in...

i run curl command again, will display 307 Temporary Redirect!

i research about 307 message says maybe have name node HA and need add other name node url to Knox WEBHDFS setting,

but this is not match my ambari system because i have only one name node (no HA),

if any setting missing in Ranger or Knox?

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

asirna — Wed, 11 Oct 2017 15:04:27 GMT

Hi @Sen Ke,

Hope you are trying to do some WEBHDFS operations. This is common for WEBHDFS.

Ex; If you want to upload a file using WEBHDFS , you have to make 2 curl calls. The response code of 1st curl call will be 307 redirect. The second call should be made to the Location Header obtained from the 1st curl call.

Sample Curl Call to Create and Write to a File

Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.

curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE
                    [&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>]
                    [&permission=<OCTAL>][&buffersize=<INT>]"

The request is redirected to a datanode where the file data is to be written:

HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE.
Content-Length: 0

Step 2: Submit another HTTP PUT request using the URL in the Location header with the file data to be written.
```
curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."
    
```
The client receives a 201 Created response with zero content length and the WebHDFS URI of the file in the Location header:
```
HTTP/1.1 201 Created
Location: webhdfs://<HOST>:<PORT>/<PATH>
Content-Length: 0
```

The above URL is for normal WEBHDFS. To access it through Knox, you can use the knox url instead of HDFS.

You can read more about WEBHDFS here and Knox WEBHDFS here

Thanks,

Aditya

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

sen_ke — Wed, 11 Oct 2017 15:38:07 GMT

Hi @Aditya Sirna

thanks your reply, but i tried as below (try to put a test file )

curl -i -k -u user1:Hadoop -X PUT 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/test?op=CREATE'

get 307 Temporary Redirect, then type

curl -i -k -u user1:Hadoop -X PUT -T /test 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/test?op=CREATE'

is it right? i get 307 again

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

asirna — Wed, 11 Oct 2017 15:44:43 GMT

@Sen Ke,

Looks like your 1st call and second call are for same url. Hit the 1st url and capture the response

Sample response

HTTP/1.1307 TEMPORARY_REDIRECT
Location: http://<xxx>:<yyy>/webhdfs/v1/<PATH>?op=CREATE.
Content-Length:0

Now make the 2nd curl call to the url obtained from the response headers of 1st call ( Location Header highlighted in bold ) (ie.. http://<xxx>:<yyy>/webhdfs/v1/<PATH>?op=CREATE...)

Thanks,

Aditya

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

asirna — Wed, 11 Oct 2017 17:10:04 GMT

Adding sample example here without knox. It would be similar with knox as well.

[root@xx user]# curl -i -X PUT "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?user.name=livy&op=CREATE"
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Tue, 26 Sep 2017 17:33:17 GMT
Date: Tue, 26 Sep 2017 17:33:17 GMT
Pragma: no-cache
Expires: Tue, 26 Sep 2017 17:33:17 GMT
Date: Tue, 26 Sep 2017 17:33:17 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: hadoop.auth="u=livy&p=livy&t=simple&e=1506483197716&s=dRvADKPG0lrenLje4fmEEdgChFw="; Path=/; HttpOnly
Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.hwx)

### second curl call with to Location obtained above
[root@xxx user]# curl -i -T /tmp/a.txt "http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false"

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

asirna — Thu, 12 Oct 2017 15:10:39 GMT

@Sen Ke, Was this issue resolved.

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

sen_ke — Thu, 12 Oct 2017 15:25:49 GMT

@Aditya Sirna

i not sure, i copied Location url to another curl command but not success , i am checking if my command wrong or other error,

i will feedback my status after test.

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

sen_ke — Fri, 13 Oct 2017 09:34:40 GMT

@Aditya Sirna

i can access webhdfs using curl now, but have a probelm

when curl via knox i only can use admin (-u admin:admin-password) and can't use other account or will reply HTTP/1.1 401 Unauthorized,

i has configured ranger and grand permission to other account (user1),

i can create/delete file with user1 in ambari file view via knox (user1 should have permission)

mycommand:

curl -i -k -u user1:Hadoop -X PUT 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/senfile1?op=CREATE'

folder permission:

drwxr-xr-x - user1 hdfs 0 2017-10-05 11:08 /user1

do you know what the problem?

Re: REST API via Knox with Ranger permission get 307 Temporary Redirect message

asirna — Fri, 13 Oct 2017 11:00:23 GMT

@Sen Ke,

Glad that it worked for you. Can you please accept the answer and start a new thread for this so that the thread will not get deviated from the main topic. Just tag me in the new thread along with the gateway.log file. I guess the user1 is not added under Advanced users-ldif in Knox advanced config.

Thanks,

Aditya