Reply
Explorer
Posts: 8
Registered: ‎04-08-2014

httpfs: can't upload file to hdfs with curl via the WebHDFS REST API?

According to the API instructions(http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_an...),i upload a textfile to hdfs,but it fail.

 

In accordance with the instructions, to carry out the two step,

but the first step(Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.)

don't return the expect result(return the datanode of the file  to be written)

 

why?

 

i have test the cdh4.4 and cdh5.0

 

 

The specific operation as follows

Blue font content is the command  of the actual implementation, but when i upload a file to hdfs,

 

 

[hdfs@localhost ~]$ curl -c ~/.httpsauth "http://192.168.230.130:14000/webhdfs/v1?op=gethomedirectory&user.name=hdfs"
{"Path":"\/user\/hdfs"}
[hdfs@localhost ~]$ curl -i -b ~/.httpfsauth -X PUT "http://192.168.230.130:14000/webhdfs/v1/test?op=CREATE&blocksize=1000&buffersize=1000"
HTTP/1.1 307 Temporary Redirect
Server: Apache-Coyote/1.1
Location: http://192.168.230.130:14000/webhdfs/v1/test?op=CREATE&buffersize=1000&blocksize=1000&data=true
Content-Type: application/json
Content-Length: 0
Date: Tue, 20 May 2014 03:52:02 GMT

[hdfs@localhost ~]$ curl -i -b ~/.httpfsauth -X PUT -T data.txt "http://192.168.230.130:14000/webhdfs/v1/test?op=CREATE"
HTTP/1.1 100 Continue

HTTP/1.1 307 Temporary Redirect
Server: Apache-Coyote/1.1
Location: http://192.168.230.130:14000/webhdfs/v1/test?op=CREATE&data=true
Content-Type: application/json
Content-Length: 0
Date: Tue, 20 May 2014 03:52:21 GMT

Posts: 307
Topics: 0
Kudos: 19
Solutions: 8
Registered: ‎08-19-2013

Re: httpfs: can't upload file to hdfs with curl via the WebHDFS REST API?

curl does not follow redirects by default.

See http://curl.haxx.se/docs/faq.html

Section 3.8 How do I tell curl to follow HTTP
redirects?

You would need to use the -L option.



David Wilder, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted
Explorer
Posts: 8
Registered: ‎04-08-2014

Re: httpfs: can't upload file to hdfs with curl via the WebHDFS REST API?

add -L option, it return 

 

HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Length: 1122
Date: Mon, 26 May 2014 07:34:55 GMT
Connection: close

 

can you give me a complete example

Explorer
Posts: 8
Registered: ‎04-08-2014

Re: httpfs: can't upload file to hdfs with curl via the WebHDFS REST API?

who can help me?