question Downloading huge results from Hue in Support Questions

Downloading huge results from Hue

Kranach — Fri, 01 Nov 2019 12:44:25 GMT

Hi,

If I run query in Hue that returns huge amount of rows, is it possible to download them through UI? I tried it using Hive query and .csv, download was succesful, but it turned out the file had exactly 100000001 rows, while actual result should be bigger. Is 100 milion some kind of limit - if so could it be lifted?

I was also thinking about storing results in HDFS and downloading them through file browser, but the problem is that when you click "save in HDFS", the whole query runs again from scratch, so effectively you need to run it twice to be able to do it (and i haven't checked if result would be stored as one file and if Hue could download it).

In short, is such a use case possible in Hue?

Re: Downloading huge results from Hue

Kranach — Wed, 28 Jan 2015 13:36:23 GMT

Errata, the file had only 1 milion lines, not 100 milions

Re: Downloading huge results from Hue

Romainr — Wed, 28 Jan 2015 14:51:36 GMT

This is https://issues.cloudera.org/browse/HUE-2142

In short right now Hue will not perform well for downloading and streaming
a lot of data to a browser as it is not designed for that.

Re: Downloading huge results from Hue

Kranach — Wed, 28 Jan 2015 15:05:41 GMT

But i dont need to see that data in a browser, i just want to download it on my PC...

Re: Downloading huge results from Hue

Romainr — Wed, 28 Jan 2015 15:14:36 GMT

The webserver is sending it to your browser, a webserver is supposed to
just send some web pages

Re: Downloading huge results from Hue

Kranach — Wed, 28 Jan 2015 15:28:24 GMT

I can download gigs of data from google drive or file hosting websites using my browser, why wouldn't it be possible here?

This means my only alternative is to tell users to install hive and tell to run something like

beeline -u jdbc:hive2://bla:10000 -n user -p password -f yourscript.q > yourresults.txt

which is a bit crap... (not to mention until Hive 13 beeline doesnt report any progress on the operation). Or let them log to my server directly and wreak havoc there 😕

All that Hue gives you already is awesome, but it needs to do more!

Re: Downloading huge results from Hue

Romainr — Wed, 28 Jan 2015 15:35:36 GMT

Please read the above JIRA for more details. Hue is only one lightweight
Python server. Google, Dropbox etc... have tens of servers dedicated to
serving files and not Web pages (the download happens from another machine).

In Hue 4 we will very probably introduce some new types of Hue servers that
will take care of this part.

Romain

Re: Downloading huge results from Hue

Kranach — Wed, 28 Jan 2015 19:23:14 GMT

I see. Maybe then there should be also some option like "execute and save to hdfs", where Hue doesnt dump results to the browser, but puts them in one file in HDFS directly? So user can get it by other means? I recently managed to store results and then download 600 MB csv file in HDFS using Hue and it kinda worked (9 milions lines, new record). Altough few minutes the service went down (not sure if because of it, or because i just started presenting Hue to my boss) so not sure if this would work.

I guess we gonna instructl users to always use LIMIT clause on their quiries, telling that this is to avoid overloading our servers (which is technically true).

Thanks for your help!

Re: Downloading huge results from Hue

Romainr — Wed, 28 Jan 2015 22:49:36 GMT

Hue has the option to save the results to HDFS and it is very scalable as
Hive is doing the writing to HDFS and then downloading from HDFS does not
require much computation from Hue.

But it indeed re-executes the SQL with the INSERT INTO /... or CREATE TABLE
AS SELECT ...

Hive or Impala does not offer a way to do both show the data in the Hue
screen and make it easy to download.

In the next version we should have some optimizations that should make more
stable to download or bump the limit.

In Hue 4 which is a big version we will tackle this as it would require a
new twin server.

So for now we recommend downloading directly from HDFS by redoing the query
for large resultsets and not bumping the 'download_row_limit' limit.

Romain

Re: Downloading huge results from Hue

Kranach — Fri, 06 Feb 2015 21:40:39 GMT

Got it. We will go this way, ironically it turned out that due to some regulatory stuff, downloading raw data from our system shouldn't bee too easy, so... we are going for good old 'it's not a bug, it's a feature' 😉

FYI, i also tried this :

beeline -u jdbc:hive2://hname:10000 -n bla -p bla -f query.q > results.txt

but it didn't do much, just hanged. Maybe hive2 (or beeline?) isn't powerful enough as well.

Thanks for all the clarifications!

Re: Downloading huge results from Hue

Romainr — Sat, 07 Feb 2015 01:56:07 GMT

Thanks for the info 🙂

Re: Downloading huge results from Hue

leandroMora — Thu, 14 Jul 2016 20:58:50 GMT

I think that the best approach to solve this issue in Hue is:

- create an external table which stores the data in TEXT format
- Load/Insert the data that you want to download there
- Go to File Browser, and browser to the location where that external table

- Download the files inside that folder

Best, Leandro

Re: Downloading huge results from Hue

SumanD — Tue, 29 Oct 2019 17:08:31 GMT

Example please for this

create an external table which stores the data in TEXT format
- Load/Insert the data that you want to download there
- Go to File Browser, and browser to the location where that external table

- Download the files inside that folder

Re: Downloading huge results from Hue

SumanD — Tue, 29 Oct 2019 17:10:23 GMT

Can you please let us know how to do for excel/csv file for the table Select * from test

beeline -u jdbc:hive2://bla:10000 -n user -p password -f yourscript.q > yourresults.txt