How to find and fix ambari stale configuration using Ambari APIs.

[#1]

###Finding Stale Configs

- We can use the Ambari APIs to find the stale configurations. (Notice: ?HostRoles/stale_configs=true)

http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=tr...

- If there is no stale configuration present in the cluster then we might see the items as empty "[]". Output may be as following:

{
  "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true",
  "items" : [ ]
}

.

- Now i intentionally made some changes to the "Advanced core-site.xml" from ambari and added the following properties to it, Just to see if it changes the stale configuration list.

hadoop.proxyuser.joy.hosts=*
hadoop.proxyuser.joy.groups=*

.

- As we made some changes and the affected components are not yet restarted hence we will see that there are few stale configuration present in the cluster then we might see the output something like following (Notice: "stale_configs" : true )

{
  "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true",
  "items" : [
    {
      "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie3.example.com/host_components/APP_TIMELINE_SERVER",
      "HostRoles" : {
        "cluster_name" : "ErieCluster",
        "component_name" : "APP_TIMELINE_SERVER",
        "host_name" : "erie3.example.com",
        "stale_configs" : true
      },
      "host" : {
        "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie3.example.com"
      }
    },
    {
      "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie1.example.com/host_components/DATANODE",
      "HostRoles" : {
        "cluster_name" : "ErieCluster",
        "component_name" : "DATANODE",
        "host_name" : "erie1.example.com",
        "stale_configs" : true
      },
      "host" : {
        "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie1.example.com"
      }
    },
    .
    .
    .======== SKIPPED some part of the output ========
    .
    {
      "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie2.example.com/host_components/ZKFC",
      "HostRoles" : {
        "cluster_name" : "ErieCluster",
        "component_name" : "ZKFC",
        "host_name" : "erie2.example.com",
        "stale_configs" : true
      },
      "host" : {
        "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie2.example.com"
      }
    }
  ]
}

.

###How to fix the Stale Config using Ambari APIs?

## Until Ambari 2.6.x ##

Now we will see how we can fix the mentioned stale configuration changes. We will need to restart the affected components in order to refresh the stale configurations. We can use the following ambari APIs (it is applicable for Ambari 2.4 and above)

# curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d  '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}' http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests

.

Example output, when the request will be successfully accepted by ambari:

OUTPUT
=======
{
  "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests/427",
  "Requests" : {
    "id" : 427,
    "status" : "Accepted"
  }
}

.

NOTE: The ["command":"RESTART"] will cause the affected services to be restarted completely (example NameNodes & DataNodes) hence before running the above command make sure that you do it in a maintenance windows (downtime expected)

## From Ambari 2.7.x onwards ##

If you are using Ambari 2.7.0 then in the API call users should also pass the "HostRoles/cluster_name=XXXXX" cluster_name property as it is mandatory.

# curl -u admin:admin -H "X-Requested-By: ambari"  -X POST -d '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true&HostRoles/cluster_name=DemoCluster"}]}' http://latest1.example.com:8080/api/v1/clusters/DemoCluster/requests

OUTPUT
========
{
  "href" : "http://latest1.example.com:8080/api/v1/clusters/DemoCluster/requests/314",
  "Requests" : {
    "id" : 314,
    "status" : "Accepted"
  }
}

.

[#2]

The API call you provided does not only start the components with stale configs, but actually all components. It has two typos, this one worked for me:

curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d  '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_components"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}' http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests

[#3]

@Jay Kumar SenSharma

Nice article. There is small correction in API to fix stale configurations:

"operation_level":"host_components" --> "operation_level":"host_component" 
and
{"hosts_predicate":"HostRoles/stale_configs:true"} --> {"hosts_predicate":"HostRoles/stale_configs=true"}

Here is the corrected API:

curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d  '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}' http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests

[#4]

@Dharmesh Jain

Thanks for sharing the findings. It was really a very keen observation. Wonderful !!!

I have incorporated the changes as you suggested above.