Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reducer Parameter on TEZ

Re: Reducer Parameter on TEZ

New Contributor

Hi Benjamin,

When i run the this parameter iam getting below error

set tez.shuffle-vertex-manager.min-src-fraction=1.0 ;

Error: Error while processing statement: Cannot modify tez.shuffle-vertex-manager.min-src-fraction at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1)

I have tez.shuffle-vertex-manager.min-src-fraction parameter already in my cluster. BUt i want to change the value for a particular job. Please help

Re: Reducer Parameter on TEZ

New Contributor

Hi Benjamin,

Any update

Re: Reducer Parameter on TEZ

New Contributor

Hi ,

When i run the this parameter iam getting below error

set tez.shuffle-vertex-manager.min-src-fraction=1.0 ;

Error: Error while processing statement: Cannot modify tez.shuffle-vertex-manager.min-src-fraction at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1)

I have tez.shuffle-vertex-manager.min-src-fraction parameter already in my cluster. BUt i want to change the value for a particular job. Please help

Re: Reducer Parameter on TEZ

New Contributor

Hi Team,.

Any clue why iam getting below error while adding the tez parameter on RUNTIME

set tez.shuffle-vertex-manager.min-src-fraction=1.0 ;

Error: Error while processing statement: Cannot modify tez.shuffle-vertex-manager.min-src-fraction at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1)

I have tez.shuffle-vertex-manager.min-src-fraction parameter already in my cluster. BUt i want to change the value for a particular job. Please help

Re: Reducer Parameter on TEZ

New Contributor

Hi!

I've tried to change this parameter already in all configs, but it does not work( Maybe some advice?

I've parameters set in hive-site.xml and in tez-site.xml, I can see then set correctly from beeline:

0: jdbc:hive2://hd-mgt06.msk.cian.ru:10000/pt> set tez.shuffle-vertex-manager.min-src-fraction;
+--------------------------------------------------+--+
|                       set                        |
+--------------------------------------------------+--+
| tez.shuffle-vertex-manager.min-src-fraction=1.0  |
+--------------------------------------------------+--+
1 row selected (0.005 seconds)
0: jdbc:hive2://hd-mgt06.msk.cian.ru:10000/pt> set tez.shuffle-vertex-manager.max-src-fraction;
+--------------------------------------------------+--+
|                       set                        |
+--------------------------------------------------+--+
| tez.shuffle-vertex-manager.max-src-fraction=1.0  |
+--------------------------------------------------+--+
1 row selected (0.008 seconds)

But when I run a job reducers are starting before mappers finished!

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ........       RUNNING     22         19        3        0       0       0
Map 5 ..........   SUCCEEDED      2          2        0        0       0       0
Map 6 ..........   SUCCEEDED      1          1        0        0       0       0
Map 7 ..........   SUCCEEDED      1          1        0        0       0       0
Reducer 2            RUNNING     49          0        6       43       0       0
Reducer 3             INITED    188          0        0      188       0       0

Reducer 4             INITED    400          0        0      400       0      
 0

--------------------------------------------------------------------------------
VERTICES: 03/07  [>>--------------------------] 3%    ELAPSED TIME: 82.90 s
--------------------------------------------------------------------------------

And when I open container logs I see that fractions is not set up correctly:

2019-02-04 11:56:22,112 [INFO] [App Shared Pool - #3] |vertexmanager.ShuffleVertexManager|: Scheduling 18 tasks for vertex: Reducer 2 with totalTasks: 388. 6 source tasks completed out of 22. SourceTaskCompletedFraction: 0.27272728 min: 0.25 max: 0.75