Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ranger services "Heartbeat lost" in Ambari, not able to stop/restart Ranger

avatar
Guru

Hello,

there is a strange situation within Ambari (v2.0.1) where Ranger suddenly lost the connectivity, since Ambari shows "Heartbeat lost" for Ranger-admin/-usersync.

The node where Ranger is/should run is fine, since it is the same as Ambari is running itself.

1836-ranger-service-heartbeat-lost.png

Trying to restart Ranger results in the following =>

1837-try-to-restart-ranger.png

and the corresponding log entries of ambari-server:

1838-ambari-server-log-error.png

What the hell has happened here ?!?!

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Gerd Koenig

Shutdown ranger-admin and user sync process

ps -ef | grep -i ranger

kill the processes if there is any

restart ambari-agent in ambari server and ranger server

then go to ambari and see if you can start ranger

if not then go back to console and restart ranger using cli

check ambari console.

View solution in original post

9 REPLIES 9

avatar
Master Mentor

@Gerd Koenig You want to restart ambari-agent in that server to see if that fixes it

avatar
Guru

Hi @Neeraj Sabharwal , unfortunately not, still the same situation. Log excerpt from ambari-agent:

1840-ambari-agent-log-after-restart.png

It is going to be late, maybe I can't see the forest but the trees ?!?! 😉

avatar
Master Mentor

@Gerd Koenig 😄 Forest and Tress..There is lion there too

On serious note: I wonder if you are hitting a bug ..see this https://issues.apache.org/jira/browse/AMBARI-12834

Now, service ranger-admin restart ..see if that help.

Need ambari log entries

avatar
Guru

@Neeraj Sabharwal , did a "sudo /etc/init.d/ranger-admin start", and verified that ranger-admin is running. But in Ambari still the same yellow sign....and in ambari logs no error, either ambari-server or -agent.

ambari-agent.log:

1851-ambari-agent-log-after-ranger-start.png

ambari-server.log (not that much entries after starting ranger-admin):

1852-ambari-server-log-after-ranger-start.png

avatar
Master Mentor

@Gerd Koenig Ok. Are you able to access Ranger UI?

avatar
Master Mentor

@Gerd Koenig

Shutdown ranger-admin and user sync process

ps -ef | grep -i ranger

kill the processes if there is any

restart ambari-agent in ambari server and ranger server

then go to ambari and see if you can start ranger

if not then go back to console and restart ranger using cli

check ambari console.

avatar
Master Guru

Hi @Gerd Koenig, can you check the status of Ranger as seen by Ambari:

curl -iv -u admin:admin -H "X-Requested-By:ambari" http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER

and component URLs

http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER/components/RANGER_ADMIN
http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/RANGER/components/RANGER_USERSYN...

The state of the service should be STARTED, and of components INSTALLED. It looks like your components are in UNKNOWN state. If so, you can go to the Ranger host page in Ambari and see can you reinstall Ranger components. I had a similar issue before, on Ranger Admin only, caused by wrong Ranger DB connection setup, and it didn't go away after fixing the DB. Had to delete the service using REST API calls and reinstall Ranger from Ambari. Hopefully you don't have to do all that.

avatar
Guru

Hi @Neeraj Sabharwal , @Predrag Minovic

thanks for your support.

Yesterday evening I did a stop of ambari-server, ambari-agent and ranger (Ranger and Ambari are on the same node), and restarted them step-by-step as mentioned in my last answer to Neeraj's post. Then I needed to have a break 😉

Looking at the situation today, showed me Ranger is back in Ambari as "known" and also showed the yellow arrows to restart due to config change. This led me to the assumption that something got messed up at config.change of Ranger and now it has been detected after the re-start of the components (and maybe waiting some time re-registering properly in Ambari). Restarting Ranger via Ambari was successfull and now everything is "green" again

Regards, Gerd

avatar
Master Mentor

@Gerd Koenig

Nice! Please pick one best answer and accept it as best answer so that we all can go home 😛