question Re: How to upload a file to HDFS using Webhdfs rest api in Archives of Support Questions (Read Only)

How to upload a file to HDFS using Webhdfs rest api

yvora — Wed, 27 Sep 2017 00:50:27 GMT

I'm looking for Wehdfs Rest api example to upload a file to HDFS. I tried with below Api but could not upload a file to hdfs

curl -i -X PUT "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?user.name=livy&op=CREATE"

curl -i -T /tmp/a.txt "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?op=CREATE&overwrite=false"

Re: How to upload a file to HDFS using Webhdfs rest api

asirna — Wed, 27 Sep 2017 01:26:35 GMT

Hi @yvora

The above API looks fine. Is the API returning 201 or any other status code. Did you make sure that /tmp/testa path exists and livy has write permission on it.

Thanks,

Aditya

Re: How to upload a file to HDFS using Webhdfs rest api

yvora — Wed, 27 Sep 2017 01:37:39 GMT

@Aditya Sirna, /tmp/testa dir is present. an livy user has permission to write to it.

I received below output while trying to run webhdfs rest api.

[root@xx user]# curl -i -X PUT "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?user.name=livy&op=CREATE"
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Tue, 26 Sep 2017 17:33:17 GMT
Date: Tue, 26 Sep 2017 17:33:17 GMT
Pragma: no-cache
Expires: Tue, 26 Sep 2017 17:33:17 GMT
Date: Tue, 26 Sep 2017 17:33:17 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: hadoop.auth="u=livy&p=livy&t=simple&e=1506483197716&s=dRvADKPG0lrenLje4fmEEdgChFw="; Path=/; HttpOnly
Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.hwx)
[root@xxx user]# curl -i -T /tmp/a.txt "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?op=CREATE&overwrite=false"
HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Tue, 26 Sep 2017 17:33:49 GMT
Date: Tue, 26 Sep 2017 17:33:49 GMT
Pragma: no-cache
Expires: Tue, 26 Sep 2017 17:33:49 GMT
Date: Tue, 26 Sep 2017 17:33:49 GMT
xRAME-OPTIONS: SAMEORIGIN

Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&namenoderpcaddress=xx:8020&createflag=&createparent=true&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.hwx)

Re: How to upload a file to HDFS using Webhdfs rest api

asirna — Wed, 27 Sep 2017 01:51:25 GMT

@yvora,

Make the 2nd curl call to the Location which was returned in the first call.

ie., http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false

Re: How to upload a file to HDFS using Webhdfs rest api

yvora — Wed, 27 Sep 2017 02:01:24 GMT

It worked. Thanks.