Cannot start secondary namenode: "currentState=STOPPING, newDesiredState=STARTED"

[#1]

Good morning!

So, I just upgraded a small cluster to Version 3.0.0. The upgrade seemed to go well. But after a reboot I am stuck- I have two services that ambari still thinks are "stopping...". (seconday namenode and also zookeeper)

Because of this, I can't get the services running. The error I get is:

Error message: java.lang.IllegalArgumentException: Invalid transition for servicecomponenthost, clusterName=ontomatedev, clusterId=2, serviceName=HDFS, componentName=SECONDARY_NAMENODE, hostname=myhost.domain.com, currentState=STOPPING, newDesiredState=STARTED

As far as I can tell, I *should* hopefully be able to "reset" this somehow via a curl command but I'm at a loss as to what.

Something like this has no effect:

curl -s -u admin:admin -H 'X-Requested-By: Ambari' -X PUT -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://localhost:8080/api/v1/clusters/ontomatedev/hosts/myhost.domain.com/host_components/SECONDARY_...

If anyone can tell me how to reset this incorrect 'state' I'd be *most* grateful!!!

[#2 · Accepted solution]

After several hours of searching, I have come to the conclusion that there is no easy fix for this, specifically through ambari API calls, which is a shame. The API calls in terms of services are limited to stopping, starting, adding, deleting. But there's no way to "fix" a broken state as per above.

Apparently the only way to solve this was to hack the ambari database and change the component state from "STOPPING" to "INSTALLED" and then I was fine.

There really oight to be a better way.....

[#3]

Thanks for the tip. I have the same problem but alas I don't have the same skills & understanding of the ambari internals.

Please tell me how you solved it -- where are the entries in the Ambari internal DB to fix, please?

Thx very much