Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

[#1]

Hello,

there is a strange situation within Ambari (v2.0.1) where Ranger suddenly lost the connectivity, since Ambari shows "Heartbeat lost" for Ranger-admin/-usersync.

The node where Ranger is/should run is fine, since it is the same as Ambari is running itself.

Trying to restart Ranger results in the following =>

and the corresponding log entries of ambari-server:

What the hell has happened here ?!?!

[#2]

@Gerd Koenig You want to restart ambari-agent in that server to see if that fixes it

[#3]

Hi @Neeraj Sabharwal , unfortunately not, still the same situation. Log excerpt from ambari-agent:

It is going to be late, maybe I can't see the forest but the trees ?!?! 😉

[#4]

@Gerd Koenig 😄 Forest and Tress..There is lion there too

On serious note: I wonder if you are hitting a bug ..see this https://issues.apache.org/jira/browse/AMBARI-12834

Now, service ranger-admin restart ..see if that help.

Need ambari log entries

[#5]

@Neeraj Sabharwal , did a "sudo /etc/init.d/ranger-admin start", and verified that ranger-admin is running. But in Ambari still the same yellow sign....and in ambari logs no error, either ambari-server or -agent.

ambari-agent.log:

ambari-server.log (not that much entries after starting ranger-admin):

[#6]

@Gerd Koenig Ok. Are you able to access Ranger UI?

[#7 · Accepted solution]

@Gerd Koenig

Shutdown ranger-admin and user sync process

ps -ef | grep -i ranger

kill the processes if there is any

restart ambari-agent in ambari server and ranger server

then go to ambari and see if you can start ranger

if not then go back to console and restart ranger using cli

check ambari console.

[#8]

Hi @Gerd Koenig, can you check the status of Ranger as seen by Ambari:

curl -iv -u admin:admin -H "X-Requested-By:ambari" http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER

and component URLs

http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER/components/RANGER_ADMIN
http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER/components/RANGER_USERSYN...

The state of the service should be STARTED, and of components INSTALLED. It looks like your components are in UNKNOWN state. If so, you can go to the Ranger host page in Ambari and see can you reinstall Ranger components. I had a similar issue before, on Ranger Admin only, caused by wrong Ranger DB connection setup, and it didn't go away after fixing the DB. Had to delete the service using REST API calls and reinstall Ranger from Ambari. Hopefully you don't have to do all that.

[#9]

Hi @Neeraj Sabharwal , @Predrag Minovic

thanks for your support.

Yesterday evening I did a stop of ambari-server, ambari-agent and ranger (Ranger and Ambari are on the same node), and restarted them step-by-step as mentioned in my last answer to Neeraj's post. Then I needed to have a break 😉

Looking at the situation today, showed me Ranger is back in Ambari as "known" and also showed the yellow arrows to restart due to config change. This led me to the assumption that something got messed up at config.change of Ranger and now it has been detected after the re-start of the components (and maybe waiting some time re-registering properly in Ambari). Restarting Ranger via Ambari was successfull and now everything is "green" again

Regards, Gerd

[#10]

@Gerd Koenig

Nice! Please pick one best answer and accept it as best answer so that we all can go home 😛