question Script to get files from HDFS to local OS "Automated process" in Archives of Support Questions (Read Only)

Script to get files from HDFS to local OS "Automated process"

nsabharwal — Tue, 29 Sep 2015 23:31:45 GMT

End user needs to get files from HDFS. The process is :

End user --> Gateway box ( Look for file locally. If not there then talk to HDFS) --> HDFS --> copy file in gateway box

Re: Script to get files from HDFS to local OS "Automated process"

andrewg — Tue, 29 Sep 2015 23:39:23 GMT

I highly recommend Knox's shell which uses a DSL for those operations http://knox.apache.org/books/knox-0-6-0/user-guide.html#WebHDFS

Great way to programmatically interact with a cluster in a controlled and audited manner (e.g. simpler DSL and secured gateway endpoint, no need to open every node's port). BTW, it's a groovy DSL, which makes it trivial to run in any Java program.

Re: Script to get files from HDFS to local OS "Automated process"

sball — Tue, 29 Sep 2015 23:41:08 GMT

An option here would be to put a standard caching web-proxy in-front of webhdfs, or of course better webhdfs through Knox, and have it ignore cache-control headers.

Re: Script to get files from HDFS to local OS "Automated process"

nsabharwal — Tue, 29 Sep 2015 23:57:56 GMT

Thanks @agrande@hortonworks.com