Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

TEZ Parameter Error on RUNTIME

avatar
Rising Star

Helloe Everyone

Iam trying to set the below parameter on HIVE using TEZ engine

set tez.shuffle-vertex-manager.min-src-fraction=1.0 ;

But getting below error:

Error: Error while processing statement: Cannot modify tez.shuffle-vertex-manager.min-src-fraction at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1)

I have tez.shuffle-vertex-manager.min-src-fraction parameter already in my cluster. But i want to change the value f. Please help

14 REPLIES 14

avatar
Rising Star

NOTE: the parameter tez.shuffle-vertex-manager.min-src-fraction I can set the parameter on HIVECLI , but its getting error while setting it on beeline . Pleae advice

avatar
Expert Contributor

Hi @suresh krish

As @Rajkumar Singh advised, that parameter cannot be modified at runtime. That means you have to modify that in hive-site.xml and restart HiveServer2. Then the change should be in effect when you connect to HS2 via beeline.

avatar
Super Guru
@suresh krish

if you look at white list param,this param(tez.shuffle-vertex-manager.min-src-fraction) is not in white list so the following piece of code(HiveConf.java) throws cannot modify exception.

public void verifyAndSet(String name, String value) throws IllegalArgumentException {
  if (modWhiteListPattern != null) {
    Matcher wlMatcher = modWhiteListPattern.matcher(name);
    if (!wlMatcher.matches()) {
      throw new IllegalArgumentException("Cannot modify " + name + " at runtime. "+ "It is not in list of params that are allowed to be modified at runtime");
    }
  }
  if (restrictList.contains(name)) {
    throw new IllegalArgumentException("Cannot modify " + name + " at runtime. It is in the list"+ "of parameters that can't be modified at runtime");
  }
  isSparkConfigUpdated = isSparkRelatedConfig(name);
  set(name, value);
}

avatar
Rising Star

So i cannot set those parameters in RUNTIME? then how do i start the reducer to only when all the mappers completes..please suggest

avatar
Expert Contributor

@suresh krish

Yes, you will not be able to set this at runtime unless you have included that in the whitelist until this is done and you have restarted hive service along with HiveServer2 this would not take effect.