[HDP-3.1][ambari-2.7.3]Changing Configs in ambari-2.7.3 fails with 'Consistency Check Failed'

[#1]

Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case.Its always suggest contact Hortonworks Support if you have trouble in your production cluster

Problem Statement: My Cluster is upgraded from ambari-2.7.1 to ambari-2.7.3 and HDP-3.0.1 to HDP-3.1. Whenever I am trying to change some configs and save it. the Below Error is shown :

Error message: </strong>Stack Advisor reported an error . Exit Code: 2. Error: KeyError: 'beeline_jdbc_url_default' 
StdOut file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.out
StdErr file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.err

and in Ambari-server.log I can see

Caused by: org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error. Exit Code: 2. Error: KeyError: 'beeline_jdbc_url_default'
StdOut file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.out
StdErr file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.err
at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.processLogs(StackAdvisorRunner.java:149)
at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.runScript(StackAdvisorRunner.java:89)
at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:314)
at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.validate(StackAdvisorHelper.java:94)
at org.apache.ambari.server.controller.internal.ValidationResourceProvider.createResources(ValidationResourceProvider.java:127)
... 105 more

And in stackadvisor.err i can see the below error :

[root@asn1 ~]# cat /var/run/ambari-server/stack-recommendations/12/stackadvisor.err
Traceback (most recent call last):
  File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 190, in <module>
    main(sys.argv)
  File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 142, in main
    result = stackAdvisor.validateConfigurations(services, hosts)
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 1079, in validateConfigurations
    validationItems = self.getConfigurationsValidationItems(services, hosts)
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 1468, in getConfigurationsValidationItems
    items.extend(self.getConfigurationsValidationItemsForService(configurations, recommendedDefaults, service, services, hosts))
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 1521, in getConfigurationsValidationItemsForService
    items.extend(serviceAdvisor.getServiceConfigurationsValidationItems(configurations, recommendedDefaults, services, hosts))
  File "/var/lib/ambari-server/resources/stacks/HDP/3.1/services/HIVE/service_advisor.py", line 143, in getServiceConfigurationsValidationItems
    return validator.validateListOfConfigUsingMethod(configurations, recommendedDefaults, services, hosts, validator.validators)
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 1491, in validateListOfConfigUsingMethod
    validationItems = method(siteProperties, siteRecommendations, configurations, services, hosts)
  File "/var/lib/ambari-server/resources/stacks/HDP/3.1/services/HIVE/service_advisor.py", line 785, in validateHiveConfigurationsEnvFromHDP30
    beeline_jdbc_url_default = hive_env["beeline_jdbc_url_default"]
KeyError: 'beeline_jdbc_url_default'

Root Cause : This issue happens due to hive-env is missing a parameter called "beeline_jdbc_url_default" which by default get added while upgrading ambari. if you have upgraded from ambari-2.7.1 to ambari-2.7.3 only then this issue will happen.

Solution: Go to Ambari-server

Execute the below command to add the missing config via configs.py (please note this cannot be added via ui)

[root@asn1 ~]# /var/lib/ambari-server/resources/scripts/configs.py -l <AMBARI_IP>  -t 8080 -u <ADMIN_USERNAME> -p <ADMIN_PASSWORD> -a set  -n <CLUSTER_NAME> -c hive-env -k beeline_jdbc_url_default -v container

for example :

[root@asn1 ~]# /var/lib/ambari-server/resources/scripts/configs.py -l asn1.openstacklocal  -t 8080 -u admin -p admin -a set  -n asnaik -c hive-env -k beeline_jdbc_url_default -v container
2018-12-20 16:21:33,820 INFO ### Performing "set":
2018-12-20 16:21:33,820 INFO ### new property - "beeline_jdbc_url_default":"container"
2018-12-20 16:21:33,835 INFO ### on (Site:hive-env, Tag:version1545035376545)
2018-12-20 16:21:33,843 INFO ### PUTting json into: doSet_version1545322893843244.json
2018-12-20 16:21:34,054 INFO ### NEW Site:hive-env, Tag:version1545322893843244

Go to Ambari-UI and restart the services are requested.

Now you can save any configs there won't be any Consistency Check Failed Error message.

[#2]

Thanks Akhil !

If your Amabri is configure with HTTPS

/var/lib/ambari-server/resources/scripts/configs.py -l asn1.openstacklocal -t 8443 -s https -u admin -p admin -a set -n asnaik -c hive-env -k beeline_jdbc_url_default -v container