question Re: Consistency check failed on Ambari in Archives of Support Questions (Read Only)

Consistency check failed on Ambari

rbalam — Mon, 19 Aug 2019 09:08:48 GMT

I installed Ambari 2.1.2.1 and HDP 2.3.2 on RH Linux 6.6. Installation went fine and services are all up and running but when I try to make any config change on Ambari UI, I see the following message.

The configuration changes could not be validated for consistency due to an unknown error. Your changes have not been saved yet. Would you like to proceed and save the changes?

How to get rid of this error?

Re: Consistency check failed on Ambari

aervits — Wed, 10 Feb 2016 02:13:16 GMT

this is the first time I see this, you might have problems in the ambari database. Can you inspect the browser console and see what commands it executes against database @rbalam

Re: Consistency check failed on Ambari

rbalam — Wed, 10 Feb 2016 02:33:43 GMT

I am using Firefox 44.0

and here is the trace from it.

TRACE: The url is: /api/v1/clusters/hwtest?fields=Clusters/health_report,Clusters/total_hosts,alerts_summary_hosts&minimal_response=true app.js:154162:7 Status code 200: Success. app.js:54173:3 App.componentsStateMapper execution time: timer started app.js:55315 App.componentsStateMapper execution time: 29.04ms app.js:55360 TRACE: The url is: /api/v1/clusters/hwtest/host_components?fields=HostRoles/component_name,HostRoles/host_name&minimal_response=true app.js:154162:7 Status code 200: Success. app.js:54173:3 TRACE: The url is: /api/v1/clusters/hwtest/configurations?type=cluster-env app.js:154162:7 Status code 200: Success. app.js:54173:3 App.componentConfigMapper execution time: timer started app.js:55076 App.componentConfigMapper execution time: 5.95ms app.js:55125

Config validation failed: Object { readyState: 4, setRequestHeader: .ajax/v.setRequestHeader(), getAllResponseHeaders: .ajax/v.getAllResponseHeaders(), getResponseHeader: .ajax/v.getResponseHeader(), overrideMimeType: .ajax/v.overrideMimeType(), abort: .ajax/v.abort(), done: f.Callbacks/p.add(), fail: f.Callbacks/p.add(), progress: f.Callbacks/p.add(), state: .Deferred/h.state(), 14 more… } error Internal Server Error Object { type: "POST", timeout: 180000, dataType: "json", statusCode: Object, headers: Object, url: "/api/v1/stacks/HDP/versions/2.3/val…", data: "{"hosts":["falbdcdq0001v.farmersinsurance.com","falbdcdq0002v.farmersinsurance.com","falbdcdq0003v.farmersinsurance.com","falbdcdq0004v.farmersinsurance.com","falbdcdq0005v.farmersinsurance.com","falbdcdq0006v.farmersinsurance.com","falbdcdq0007v.farmersinsurance.com"],"services":["AMBARI_METRICS","ATLAS","FALCON","FLUME","HBASE","HDFS","HIVE","KAFKA","KNOX","MAPREDUCE2","OOZIE","PIG","RANGER","RANGER_KMS","SPARK","SQOOP","STORM","TEZ","YARN","ZOOKEEPER"],"validate":"configurations","recommendations":{"blueprint":{"host_groups":[{"name":"host-group-1","components":[{"name":"FALCON_CLIENT"},{"name":"HBASE_CLIENT"},{"name":"HDFS_CLIENT"},{"name":"HCAT"},{"name":"HIVE_CLIENT"},{"name":"MAPREDUCE2_CLIENT"},{"name":"OOZIE_CLIENT"},{"name":"PIG"},{"name":"SPARK_CLIENT"},{"name":"SQOOP"},{"name":"TEZ_CLIENT"},{"name":"YARN_CLIENT"},{"name":"ZOOKEEPER_CLIENT"},{"name":"METRICS_MONITOR"},{"name":"FALCON_CLIENT"},{"name":"HBASE_CLIENT"},{"name":"HCAT"},{"name":"HDFS_CLIENT"},{"name":"HIVE_CLIENT"[…], context: Object, beforeSend: ajax<.send/opt.beforeSend(), success: ajax<.send/opt.success(), 2 more… } app.js:64175:5 Error code 500: Internal Error on server side. app.js:54195:3 Config validation failed. Going ahead with saving of configs app.js:64188:7 TRACE: The url is: /api/v1/clusters?fields=Clusters/provisioning_state app.js:154162:7 Status code 200: Success. app.js:54173:3 TRACE: The url is: /api/v1/clusters/hwtest/requests?to=end&page_size=10&fields=Requests app.js:154162:7 Status code 200: Success. app.js:54173:3

Re: Consistency check failed on Ambari

nsabharwal — Wed, 10 Feb 2016 02:34:04 GMT

@rbalam

What happens when you hit proceed? and while doing that do tail -f on /var/log/ambari-server/ambari-server.log

Re: Consistency check failed on Ambari

aervits — Wed, 10 Feb 2016 02:35:44 GMT

@rbalam

try restarting ambari server and agents, then try again. HTTP code 500 means a problem on the server side.

Re: Consistency check failed on Ambari

nsabharwal — Wed, 10 Feb 2016 02:36:03 GMT

@rbalam See this thread http://stackoverflow.com/questions/4789613/ajax-500-internal-server-error

Re: Consistency check failed on Ambari

nsabharwal — Wed, 10 Feb 2016 02:42:39 GMT

@rbalam Clean the browser cache and refresh the ambari console to see the behavior before any other change.

Re: Consistency check failed on Ambari

nsabharwal — Wed, 10 Feb 2016 03:45:41 GMT

@rbalam You may want to check few settings on permissions...I had the same issue with this

[root@sandbox ~]# ls -l /etc/login.defs

-rw-r--r--. 1 root root 1816 2015-03-27 13:14 /etc/login.defs

[root@sandbox ~]#

Re: Consistency check failed on Ambari

lvazquez — Thu, 13 Sep 2018 08:29:36 GMT

I have experienced this problem after changing Ambari Server to run as a non privileged user "ambari-server". In my case I can see in the logs (/var/log/ambari-server/ambari-server.log) the following:

12 Sep 2018 22:06:57,515 ERROR [ambari-client-thread-6396] BaseManagementHandler:61 - Caught a system exception while attempting to create a resource: Error occured during stack advisor command invocation: Cannot create /var/run/ambari-server/stack-recommendations

This error happens because in CentOS/RedHat /var/run is really a symlink to /run which is a tmpfs filesystem mounted at boot time from RAM. So if I manually create the folder with the required privileges it won't survive a reboot and because the unprivileged user running Ambari Server is unable to create the required directory the error occurs.

I was able to partially fix this using systemd-tmpfiles feature by creating a file /etc/tmpfiles.d/ambari-server.conf with following content:

d /run/ambari-server 0775 ambari-server hadoop -
d /run/ambari-server/stack-recommendations 0775 ambari-server hadoop -

With this file in place running "systemd-tmpfiles --create" will create the folders with the required privileges. According to the following RedHat documentation this should be automagically run at boot time to setup everything:

https://developers.redhat.com/blog/2016/09/20/managing-temporary-files-with-systemd-tmpfiles-on-rhel7/

However sometimes this doesn't happens (I don't know why) and I have to run the previous command manually to fix this error.