REST API via Knox with Ranger permission get 307 Temporary Redirect message

[#1]

Hi All, to fix knox rest api 404 not found error, i modify WEBHDFS url from {{webhdfs_service_urls}} to http://My_NameNode:50070/webhdfs, it's worked.

then i using Ranger to control HDFS access permission, and configure Ranger HDFS and Knox plug-in...

i run curl command again, will display 307 Temporary Redirect!

i research about 307 message says maybe have name node HA and need add other name node url to Knox WEBHDFS setting,

but this is not match my ambari system because i have only one name node (no HA),

if any setting missing in Ranger or Knox?

[#2 · Accepted solution]

Hi @Sen Ke,

Hope you are trying to do some WEBHDFS operations. This is common for WEBHDFS.

Ex; If you want to upload a file using WEBHDFS , you have to make 2 curl calls. The response code of 1st curl call will be 307 redirect. The second call should be made to the Location Header obtained from the 1st curl call.

Sample Curl Call to Create and Write to a File

• Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE
                      [&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>]
                      [&permission=<OCTAL>][&buffersize=<INT>]"
      

  The request is redirected to a datanode where the file data is to be written:

  HTTP/1.1 307 TEMPORARY_REDIRECT
  Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE.
  Content-Length: 0
      

• Step 2: Submit another HTTP PUT request using the URL in the Location header with the file data to be written.

  curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."
      

  The client receives a 201 Created response with zero content length and the WebHDFS URI of the file in the Location header:

  HTTP/1.1 201 Created
  Location: webhdfs://<HOST>:<PORT>/<PATH>
  Content-Length: 0

The above URL is for normal WEBHDFS. To access it through Knox, you can use the knox url instead of HDFS.

You can read more about WEBHDFS here and Knox WEBHDFS here

Thanks,

Aditya

[#3]

Hi @Aditya Sirna

thanks your reply, but i tried as below (try to put a test file )

curl -i -k -u user1:Hadoop -X PUT 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/test?op=CREATE'

get 307 Temporary Redirect, then type

curl -i -k -u user1:Hadoop -X PUT -T /test 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/test?op=CREATE'

is it right? i get 307 again

[#4]

@Sen Ke,

Looks like your 1st call and second call are for same url. Hit the 1st url and capture the response

Sample response

1. HTTP/1.1307 TEMPORARY_REDIRECT
2. Location: http://<xxx>:<yyy>/webhdfs/v1/<PATH>?op=CREATE.
3. Content-Length:0

Now make the 2nd curl call to the url obtained from the response headers of 1st call ( Location Header highlighted in bold ) (ie.. http://<xxx>:<yyy>/webhdfs/v1/<PATH>?op=CREATE...)

Thanks,

Aditya

[#5]

Adding sample example here without knox. It would be similar with knox as well.

[root@xx user]# curl -i -X PUT "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?user.name=livy&op=CREATE"
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Tue, 26 Sep 2017 17:33:17 GMT
Date: Tue, 26 Sep 2017 17:33:17 GMT
Pragma: no-cache
Expires: Tue, 26 Sep 2017 17:33:17 GMT
Date: Tue, 26 Sep 2017 17:33:17 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: hadoop.auth="u=livy&p=livy&t=simple&e=1506483197716&s=dRvADKPG0lrenLje4fmEEdgChFw="; Path=/; HttpOnly
Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&cre...
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.hwx)

### second curl call with to Location obtained above
[root@xxx user]# curl -i -T /tmp/a.txt "http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false"

[#6]

@Sen Ke, Was this issue resolved.

[#7]

@Aditya Sirna

i not sure, i copied Location url to another curl command but not success , i am checking if my command wrong or other error,

i will feedback my status after test.

[#8]

@Aditya Sirna

i can access webhdfs using curl now, but have a probelm

when curl via knox i only can use admin (-u admin:admin-password) and can't use other account or will reply HTTP/1.1 401 Unauthorized,

i has configured ranger and grand permission to other account (user1),

i can create/delete file with user1 in ambari file view via knox (user1 should have permission)

mycommand:

curl -i -k -u user1:Hadoop -X PUT 'https://knoxHost:8443/gateway/default/webhdfs/v1/user1/senfile1?op=CREATE'

folder permission:

drwxr-xr-x - user1 hdfs 0 2017-10-05 11:08 /user1

do you know what the problem?

[#9]

@Sen Ke,

Glad that it worked for you. Can you please accept the answer and start a new thread for this so that the thread will not get deviated from the main topic. Just tag me in the new thread along with the gateway.log file. I guess the user1 is not added under Advanced users-ldif in Knox advanced config.

Thanks,

Aditya