question how to call the web-service to read/write the data from the HDFS in Archives of Support Questions (Read Only)

how to call the web-service to read/write the data from the HDFS

balajibobba — Thu, 18 Feb 2016 00:09:11 GMT

Since Apache Drill not supported as of now, i need to find out the way to call web-service

Re: how to call the web-service to read/write the data from the HDFS

aervits — Thu, 18 Feb 2016 00:10:53 GMT

WebHCat

# this will execute a hive query and save result to hdfs file in your home directory called output
curl -s -d execute="select+*+from+sample_08;" \
       -d statusdir="output" \
       'http://localhost:50111/templeton/v1/hive?user.name=root'
       
# if you ls on the directory, it will have two files, stderr and stdout
hdfs dfs -ls output

# if the job succeeded, you can cat the stdout file and view the results
hdfs dfs -cat output/stdout       

WebHDFS

# list the output directory, notice the webhdfs port
curl -i "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/?op=LISTSTATUS"

# read the output file
curl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/stdout?op=OPEN"

# rename a file, if you get dr. who error, add &user.name=root or any other user in the context
curl -i -X PUT "sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/stdout?op=RENAME&user.name=root&destination=/user/root/newname"

# read the output of the new file
curl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/newname?op=OPEN"

Re: how to call the web-service to read/write the data from the HDFS

bleonhardi — Thu, 18 Feb 2016 00:13:17 GMT

So you want to read/write files in HDFS?

This is what webhdfs is for:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

Re: how to call the web-service to read/write the data from the HDFS

balajibobba — Thu, 18 Feb 2016 00:15:04 GMT

Thanks Artem, but I may have to use SOAP/REST from another client to read or write the data into HDFS... so any idea about that?

Re: how to call the web-service to read/write the data from the HDFS

aervits — Thu, 18 Feb 2016 00:20:28 GMT

you can use the hdfs java api and wrap your rest code? http://tutorials.techmytalk.com/2014/08/16/hadoop-hdfs-java-api/ @Balaji Bobba

Re: how to call the web-service to read/write the data from the HDFS

balajibobba — Thu, 18 Feb 2016 00:25:49 GMT

Thanks Benjamin, I will look into this.

Re: how to call the web-service to read/write the data from the HDFS

balajibobba — Thu, 18 Feb 2016 00:26:15 GMT

Thanks, this may work I will look into this.

Re: how to call the web-service to read/write the data from the HDFS

aervits — Thu, 18 Feb 2016 00:27:28 GMT

@Balaji Bobba you're welcome, try it out and post your solution.