Support Questions

Find answers, ask questions, and share your expertise

Curl command- Connected httpfs/webhdfs but not getting output

avatar

Ran below curl commands, able to connected but not getting results. Just its throwong letter P along with special characters.
Env is Cloudera 6.1.2.
Want to use httpfs to put the files in HDFS.
Please suggest.

 

curl -i -X PUT -T /home/CORP/isuy/Ganesh.txt "http://aalhauap2u01:14000/hdfs:////user/isuy/testfiles?op=CREATE"

curl -i "http://aalhauap2g01:14000/webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS"

curl -i "http://aalhauap2u02:14000/webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS" -v
* About to connect() to aalhauap2u02 port 14000 (#0)
* Trying 10.91.23.152...
* Connected to aalhauap2u02 (10.91.23.152) port 14000 (#0)
> GET /webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS HTTP/1.1
> User-Agent: curl/7.29.0
> Host: aalhauap2u02:14000
> Accept: */*
>
* Connection #0 to host aalhauap2u02 left intact


Thanks,

2 REPLIES 2

avatar
Contributor

Hi Sree,

 

I'm not entirely sure of the syntax you're using for your curl request.

The HttpFS role still uses the same API syntax as the WebHDFS API.

 

If you haven't already, I would suggest reviewing the API method for creating and writing to a file via the WebHDFS API[1]. 

Your command, when executed correctly, should look similar to the following:

curl -i -k -X PUT "https://your.host.com:14000/webhdfs/v1/tmp/file.txt?op=CREATE"

 

The HDFS service should then spit back a URL to follow through to actually upload your file to. In the above example, that returned looked like this:

HTTP/1.1 307 Temporary Redirect
Date: Thu, 18 Apr 2019 22:23:31 GMT
Cache-Control: no-cache
Expires: Thu, 18 Apr 2019 22:23:31 GMT
Date: Thu, 18 Apr 2019 22:23:31 GMT
Pragma: no-cache
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
WWW-Authenticate: Negotiate YGwGCSqGSIb3EgECAgIAb10wW6ADAgEFoQMCAQ+iTzBNoAMCARCiRgRE/UEMlOPP/tGaYJaWJMk3AyJMlMu9cHklguw/oEKRQwULVYvOiRhJAaTHo8FL9x3Lqhn/F4XpCscNa8IWg/LCflyYci8=
Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@YOUR.DOMAIN.COM&t=kerberos-dt&e=1555662211059&s=TWQoO1FJ/fNNEHxLTNOyIivzsEieJ46M1oUv+DzB7OY="; Path=/; Secure; HttpOnly
Location: https://your.host.com:14000/webhdfs/v1/tmp/file.txt?op=CREATE&data=true             <======*THIS IS THE URL WE WRITE TO*
Content-Type: application/json;charset=utf-8
Content-Length: 0

 

At that point, we actually submit the payload against the returned path from HDFS:

curl -i -k -H "Content-Type:application/octet-stream" -X PUT -T file.txt "https://your.host.com:14000/webhdfs/v1/tmp/file.txt?op=CREATE&data=true"

And the data shows up:

hdfs dfs -cat /tmp/file.txt
>>this is a file

 

 

[1] https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_...

avatar

Thank you, it helps