Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

Solved Go to solution
Highlighted

Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

Guru

Hello,

there is a strange situation within Ambari (v2.0.1) where Ranger suddenly lost the connectivity, since Ambari shows "Heartbeat lost" for Ranger-admin/-usersync.

The node where Ranger is/should run is fine, since it is the same as Ambari is running itself.

1836-ranger-service-heartbeat-lost.png

Trying to restart Ranger results in the following =>

1837-try-to-restart-ranger.png

and the corresponding log entries of ambari-server:

1838-ambari-server-log-error.png

What the hell has happened here ?!?!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

@Gerd Koenig

Shutdown ranger-admin and user sync process

ps -ef | grep -i ranger

kill the processes if there is any

restart ambari-agent in ambari server and ranger server

then go to ambari and see if you can start ranger

if not then go back to console and restart ranger using cli

check ambari console.

9 REPLIES 9

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

@Gerd Koenig You want to restart ambari-agent in that server to see if that fixes it

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

Guru

Hi @Neeraj Sabharwal , unfortunately not, still the same situation. Log excerpt from ambari-agent:

1840-ambari-agent-log-after-restart.png

It is going to be late, maybe I can't see the forest but the trees ?!?!

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

@Gerd Koenig :D Forest and Tress..There is lion there too

On serious note: I wonder if you are hitting a bug ..see this https://issues.apache.org/jira/browse/AMBARI-12834

Now, service ranger-admin restart ..see if that help.

Need ambari log entries

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

Guru

@Neeraj Sabharwal , did a "sudo /etc/init.d/ranger-admin start", and verified that ranger-admin is running. But in Ambari still the same yellow sign....and in ambari logs no error, either ambari-server or -agent.

ambari-agent.log:

1851-ambari-agent-log-after-ranger-start.png

ambari-server.log (not that much entries after starting ranger-admin):

1852-ambari-server-log-after-ranger-start.png

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

@Gerd Koenig Ok. Are you able to access Ranger UI?

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

@Gerd Koenig

Shutdown ranger-admin and user sync process

ps -ef | grep -i ranger

kill the processes if there is any

restart ambari-agent in ambari server and ranger server

then go to ambari and see if you can start ranger

if not then go back to console and restart ranger using cli

check ambari console.

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

Hi @Gerd Koenig, can you check the status of Ranger as seen by Ambari:

curl -iv -u admin:admin -H "X-Requested-By:ambari" http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER

and component URLs

http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER/components/RANGER_ADMIN
http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER/components/RANGER_USERSYN...

The state of the service should be STARTED, and of components INSTALLED. It looks like your components are in UNKNOWN state. If so, you can go to the Ranger host page in Ambari and see can you reinstall Ranger components. I had a similar issue before, on Ranger Admin only, caused by wrong Ranger DB connection setup, and it didn't go away after fixing the DB. Had to delete the service using REST API calls and reinstall Ranger from Ambari. Hopefully you don't have to do all that.

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

Guru

Hi @Neeraj Sabharwal , @Predrag Minovic

thanks for your support.

Yesterday evening I did a stop of ambari-server, ambari-agent and ranger (Ranger and Ambari are on the same node), and restarted them step-by-step as mentioned in my last answer to Neeraj's post. Then I needed to have a break ;)

Looking at the situation today, showed me Ranger is back in Ambari as "known" and also showed the yellow arrows to restart due to config change. This led me to the assumption that something got messed up at config.change of Ranger and now it has been detected after the re-start of the components (and maybe waiting some time re-registering properly in Ambari). Restarting Ranger via Ambari was successfull and now everything is "green" again

Regards, Gerd

Re: Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

@Gerd Koenig

Nice! Please pick one best answer and accept it as best answer so that we all can go home :P

Don't have an account?
Coming from Hortonworks? Activate your account here