04-16-2019 07:14 PM
Ran below curl commands, able to connected but not getting results. Just its throwong letter P along with special characters.
Env is Cloudera 6.1.2.
Want to use httpfs to put the files in HDFS.
curl -i -X PUT -T /home/CORP/isuy/Ganesh.txt "http://aalhauap2u01:14000/hdfs:////user/isuy/testfiles?op=CREATE"
curl -i "http://aalhauap2g01:14000/webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS"
curl -i "http://aalhauap2u02:14000/webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS" -v
* About to connect() to aalhauap2u02 port 14000 (#0)
* Trying 10.91.23.152...
* Connected to aalhauap2u02 (10.91.23.152) port 14000 (#0)
> GET /webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS HTTP/1.1
> User-Agent: curl/7.29.0
> Host: aalhauap2u02:14000
> Accept: */*
* Connection #0 to host aalhauap2u02 left intact
04-18-2019 03:31 PM
I'm not entirely sure of the syntax you're using for your curl request.
The HttpFS role still uses the same API syntax as the WebHDFS API.
If you haven't already, I would suggest reviewing the API method for creating and writing to a file via the WebHDFS API.
Your command, when executed correctly, should look similar to the following:
curl -i -k -X PUT "https://your.host.com:14000/webhdfs/v1/tmp/file.txt?op=CREATE"
The HDFS service should then spit back a URL to follow through to actually upload your file to. In the above example, that returned looked like this:
HTTP/1.1 307 Temporary Redirect Date: Thu, 18 Apr 2019 22:23:31 GMT Cache-Control: no-cache Expires: Thu, 18 Apr 2019 22:23:31 GMT Date: Thu, 18 Apr 2019 22:23:31 GMT Pragma: no-cache X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block WWW-Authenticate: Negotiate YGwGCSqGSIb3EgECAgIAb10wW6ADAgEFoQMCAQ+iTzBNoAMCARCiRgRE/UEMlOPP/tGaYJaWJMk3AyJMlMu9cHklguw/oEKRQwULVYvOiRhJAaTHo8FL9x3Lqhn/F4XpCscNa8IWg/LCflyYci8= Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@YOUR.DOMAIN.COM&t=kerberos-dt&e=1555662211059&s=TWQoO1FJ/fNNEHxLTNOyIivzsEieJ46M1oUv+DzB7OY="; Path=/; Secure; HttpOnly Location: https://your.host.com:14000/webhdfs/v1/tmp/file.txt?op=CREATE&data=true <======*THIS IS THE URL WE WRITE TO* Content-Type: application/json;charset=utf-8 Content-Length: 0
At that point, we actually submit the payload against the returned path from HDFS:
curl -i -k -H "Content-Type:application/octet-stream" -X PUT -T file.txt "https://your.host.com:14000/webhdfs/v1/tmp/file.txt?op=CREATE&data=true"
And the data shows up:
hdfs dfs -cat /tmp/file.txt >>this is a file