question Re: Downloading a file inside a hadoop archive using Apache Knox in Archives of Support Questions (Read Only)

Downloading a file inside a hadoop archive using Apache Knox

pooja_khandelwa — Tue, 21 Jun 2016 13:19:16 GMT

I am able to download a hdfs file /org/project/archived/data/hive/warehouse/Stats/2016_06_20.txt in a broswer thorugh knox using below URL

http://hostname:8443/knox/nm1/webhdfs/v1/org/project/archived/data/hive/warehouse/Stats/2016_06_20.txt?op=OPEN

Now I have a file in a hadoop archive as below.

har:///org/project/archived/data/hive/warehouse/test.har/Stats/2016_06_20.txt

How can i do the same for the above file?

Re: Downloading a file inside a hadoop archive using Apache Knox

SK1 — Tue, 21 Jun 2016 13:49:32 GMT

@pooja khandelwal: You may use following approach to get hadoop archived file in your local machine.

hadoop fs -text har:///org/project/archived/data/hive/warehouse/test.har/Stats/2016_06_20.txt > 2016_06_20.txt

Re: Downloading a file inside a hadoop archive using Apache Knox

pooja_khandelwa — Tue, 21 Jun 2016 14:06:54 GMT

I want to download the file through browser only.

Re: Downloading a file inside a hadoop archive using Apache Knox

pminovic — Tue, 21 Jun 2016 14:17:22 GMT

First download the har's _index file located at /org/project/archived/data/hive/warehouse/test.har/_index. Then locate Stats/2016_06_20.txt in _index and its data-n file, the offset within the data file and its length. Suppose it's in data-0 and offset=125000 and file-length=8200, then you can access

http://hostname:8443/knox/nm1/webhdfs/v1/org/project/archived/data/hive/warehouse/test.har/data-0?op=OPEN&offset=125000&length=8200

Check this nicely written blog for a full example and a PHP script which can automate the process.

Re: Downloading a file inside a hadoop archive using Apache Knox

pooja_khandelwa — Wed, 22 Jun 2016 10:10:07 GMT

Thank you.