Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can't Increase number of TEZ Mappers in HiveServer2 Interactive

Can't Increase number of TEZ Mappers in HiveServer2 Interactive

New Contributor

Size of single file is 2.4GB, number of records in table more than 22 millions. My cluster is on HDP 2.6 and consists of 16 nodes (each with 96GB memory).

I can't increase the number of mappers. I set followings:

set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;

set tez.grouping.min-size=16777216; (16MB)

set tez.grouping.max-size=107374182; (107MB)

set hive.optimize.index.filter=true;

set use.hive.interactive.mode=true;

I'm executing following query "select PropertyType, count(*) as count from houses group by PropertyType;"

but each time Hive TEZ creates just ONE Mapper, thats why this query takes too long (135 sec., 95% of time spent for mapping task). I'm using Beeline interface, but even Hive CLI has same result.

----------------------------------------------------------------------------

INFO  : Dag name: select PropertyType, count(*)...PropertyType(Stage-1)
INFO  : Status: Running (Executing on YARN cluster with App id application_1521676736844_0019)


--------------------------------------------------------------------------------
        VERTICES        MODE  STATUS      TOTAL  COMPLETED  RUNNING  PENDING  FAILED
--------------------------------------------------------------------------------
Map 1 ..........        llap  SUCCEEDED          1        1        0       0       0
Reducer 2 ......        llap  SUCCEEDED         59       59        0       0       0
--------------------------------------------------------------------------------
VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 131.76 s   
--------------------------------------------------------------------------------
INFO  : Status: DAG finished successfully in 131.65 seconds
INFO  : 
INFO  : Query Execution Summary
INFO  : ----------------------------------------------------------------------------------------------
INFO  : OPERATION                            DURATION
INFO  : ----------------------------------------------------------------------------------------------
INFO  : Compile Query                           0.95s
INFO  : Prepare Plan                            0.38s
INFO  : Submit Plan                             0.36s
INFO  : Start DAG                               0.47s
INFO  : Run DAG                               131.65s
INFO  : ----------------------------------------------------------------------------------------------
INFO  : ...
INFO  : OK
...

So, how can I increase the number of Mapper tasks?

I'd highly appreciate any help

1 REPLY 1
Highlighted

Re: Can't Increase number of TEZ Mappers in HiveServer2 Interactive

What is the file type?

Don't have an account?
Coming from Hortonworks? Activate your account here