HIVE: An application TEZ appears when I run with MR engine

[#1]

Hi,

When I run an oozie hive action with "SET hive.execution.engine=mr;" I see in Resource Manager UI 2 applications :

One application that corresponds to my action HIVE (insert into hdfs...) with the name i fixed and application Type = Map reduce and another application with the name like "HIVE-dhhdgdgjdg....) and a type of engine is TEZ !!!

I note that the default engine on my cluster HDP is TEZ and I just change on Map reduce in my Oozie Action:

SET hive.execution.engine=mr;
INSERT OVERWRITE TABLE hdfs.tab1 SELECT a,b,c FROM cassandra.tab1;

Have you any idea about the source of the second TEZ application ?

[#2]

Can you set this property in the configuration tag of the hive action and try out. That should work.

[#3]

Hi @Abhishek Bafna,

My action works properly, I want to understand why my Oozie action creates 2 application : One with MR engine and another with TEZ engine.

[#4]

@Ahmed ELJAMI

Each Oozie action, it encapsulated in Ooize container. Once the oozie action is triggered, 1st the container triggers with oozie configured settings and then the actual executable with overridden settings. This is default behavior of Oozie.

Thus, in your case, Oozie Hive containter was triggered 1st, with TEZ engine. Later when the Hive script got executed, the setting was overridden with MapReduce.

Is this causing any issue/side-effect?

[#5]

@Dhaval Modi

In my case, I set the queue for Oozie container with the following config:

oozieLauncherQueue=oozie-launcher jobQueue=default

I can see the Oozie Container in the Queue oozie-launcher :

userA oozie:launcher:T=shell:W=snapshot_userA:A=copy_data:ID=0000014-160905162411567-oozie-oozi-W

In the queue default, I see 2 applications that I mentioned above, when I change the default Engine on HDP (MR), I see one application on the default queue.

>Is this causing any issue/side-effect?

The application Hive-XXXX occupies every time an container without doing anything.

[#6]

@Ahmed ELJAMI

This is default behavior for HIVE action and its drawback for cluster with less no. of containers.

[#7]

Ok :)

why when I change the default engine to MR in HDP, I don't see anymore the 3rd application (TEZ) ?

Oozie container with TEZ is created only when the default engine is TEZ even if I do not use it ?

Thx