TEZ Parameter Error on RUNTIME

[#1]

Helloe Everyone

Iam trying to set the below parameter on HIVE using TEZ engine

set tez.shuffle-vertex-manager.min-src-fraction=1.0 ;

But getting below error:

Error: Error while processing statement: Cannot modify tez.shuffle-vertex-manager.min-src-fraction at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1)

I have tez.shuffle-vertex-manager.min-src-fraction parameter already in my cluster. But i want to change the value f. Please help

[#2]

NOTE: the parameter tez.shuffle-vertex-manager.min-src-fraction I can set the parameter on HIVECLI , but its getting error while setting it on beeline . Pleae advice

[#3]

Hi @suresh krish

As @Rajkumar Singh advised, that parameter cannot be modified at runtime. That means you have to modify that in hive-site.xml and restart HiveServer2. Then the change should be in effect when you connect to HS2 via beeline.

[#4]

@suresh krish

if you look at white list param,this param(tez.shuffle-vertex-manager.min-src-fraction) is not in white list so the following piece of code(HiveConf.java) throws cannot modify exception.

public void verifyAndSet(String name, String value) throws IllegalArgumentException {
  if (modWhiteListPattern != null) {
    Matcher wlMatcher = modWhiteListPattern.matcher(name);
    if (!wlMatcher.matches()) {
      throw new IllegalArgumentException("Cannot modify " + name + " at runtime. "+ "It is not in list of params that are allowed to be modified at runtime");
    }
  }
  if (restrictList.contains(name)) {
    throw new IllegalArgumentException("Cannot modify " + name + " at runtime. It is in the list"+ "of parameters that can't be modified at runtime");
  }
  isSparkConfigUpdated = isSparkRelatedConfig(name);
  set(name, value);
}

[#5]

So i cannot set those parameters in RUNTIME? then how do i start the reducer to only when all the mappers completes..please suggest

[#6]

@suresh krish

Yes, you will not be able to set this at runtime unless you have included that in the whitelist until this is done and you have restarted hive service along with HiveServer2 this would not take effect.