question Hive's taking to much time. It is normal? in Support Questions

Hive's taking to much time. It is normal?

cristianvlean — Mon, 19 Aug 2019 08:56:35 GMT

Hi,

I'm trying to do a select count(*) from tweets; using Hive CLI. But as you can see, it is taking to much time (i guess)

It is normal?

Any tip?

Regards

Re: Hive's taking to much time. It is normal?

jpp — Sat, 03 Dec 2016 01:54:28 GMT

The Tez job has not started: both the mapper and the reducer are in "pending" state and haven't yet been launched. Once launched they would enter "running" state.

Check yarn to ensure there is enough room in your queue to fit the containers (http://sandbox:8088). There isn't much ram in a sandbox, and it could all be taken up by a Spark instance or by the Tez instances of hiveserver2.

Re: Hive's taking to much time. It is normal?

cristianvlean — Mon, 19 Aug 2019 08:56:27 GMT

Jean-Phillippe, as you may guess, i am new with the Sandbox. How can i know if there's enough room in my queue to fit containers?

These are my setting.

Re: Hive's taking to much time. It is normal?

zyang — Sat, 03 Dec 2016 02:22:27 GMT

Job is not started at all. Please share more info for debugging(cluster size, query, data size, hive log, etc)

Re: Hive's taking to much time. It is normal?

cristianvlean — Sat, 03 Dec 2016 02:43:04 GMT

Hi Zyang,

Can you tell me where can i find that information?

Regards

Re: Hive's taking to much time. It is normal?

zyang — Sat, 03 Dec 2016 03:00:21 GMT

Cluster size: yarn node -list

Hive log: /tmp/<username>/hive.log

Re: Hive's taking to much time. It is normal?

zyang — Sat, 03 Dec 2016 03:04:30 GMT

Never mind about cluster size. I saw you are using sandbox. Probably job cannot be started because there is not enough resource. You can try allocate more memory for VM or decrease tez container size.

Re: Hive's taking to much time. It is normal?

cristianvlean — Sat, 03 Dec 2016 03:10:39 GMT

Zyang, what is the purpose about decreasing Tez Container Size?

Re: Hive's taking to much time. It is normal?

cristianvlean — Sat, 03 Dec 2016 03:43:46 GMT

Well Zyang, as you suggested i tried decreasing tez container size.

Now i got the following error

Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1480577643003_0005_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1480577643003_0005_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1480577643003_0005_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

Any suggestion?

regards

Re: Hive's taking to much time. It is normal?

zyang — Sat, 03 Dec 2016 08:02:37 GMT

Your resource is limited. To solve the problem, either you allocate more resource or just request ask fro resource.

Re: Hive's taking to much time. It is normal?

zyang — Sat, 03 Dec 2016 08:06:26 GMT

Map vertex get killed because Reduce vertex failed. Any log for reduce vertex?

Re: Hive's taking to much time. It is normal?

jpp — Wed, 07 Dec 2016 00:55:41 GMT

Since you are in a sandbox, you need to reduce the amount of memory taken by each component so that they can fit (try running with 12 GB if you can). Reduce the memory footprint as follows:

Tez container size = 1024 MB
Map join, per Map memory = 256 MB
Metastore heap = 512 MB
Client heap = 512 MB
Tez Client->tez.am.resource.memory.mb = 512

YARN will need to fit at least one Tez AM (512 MB) and a couple Tez containers (512MB *2). You can check how much memory is allocated to YARN on the YARN config page "Memory allocated for all YARN containers on a node".

Re: Hive's taking to much time. It is normal?

cristianvlean — Wed, 07 Dec 2016 11:08:39 GMT

Dead Jean-Phillipe.

I tried to follow your instructions. But, as you can see is giving me headaches

"YARN will need to fit at least one Tez AM (512 MB) and a couple Tez containers (512MB *2). You can check how much memory is allocated to YARN on the YARN config page "Memory allocated for all YARN containers on a node"."

Can you please guide me please.

Regards

Re: Hive's taking to much time. It is normal?

jpp — Fri, 16 Dec 2016 08:56:38 GMT

Hi Cristian, the amount of memory that yarn can allocate is controlled by the setting "Memory allocated for all YARN containers on a node" under YARN. Set this to 3 GB, and that should give enough room for Tez to run (it needs 2.5 GB if you follow the settings above.).

Re: Hive's taking to much time. It is normal?

cristianvlean — Wed, 21 Dec 2016 11:01:01 GMT

I've placed more ram, added jar path missing and worked perfect.

Thanks for your help