Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase fails to start after graceful shutdown until Keytabs are regenerated

HBase fails to start after graceful shutdown until Keytabs are regenerated

Explorer

Greetings,

 

We're encountering a rather strange issue. We are running HBase as part of a healthy CDH 5.15 cluster. Because of configuration changes we have to restart HBase every now and then.

 

Every time, no matter what, HBase fails to start after a graceful shutdown. The error is always the same, kinit fails. I have to manually regenerate the HBase keytabs from cloudera manager, afterwards HBase starts right away.

 

I looked around online but failed to find any mention of such behavior. Other services handle restarts without hiccups, no need to regenerate other keytabs.

 

Please let me know if additional details are needed to further debug this issue.

 

Thank you for your help in advance!

 

Kind regards,

Julius

4 REPLIES 4

Re: HBase fails to start after graceful shutdown until Keytabs are regenerated

Super Guru

@matagyula,

 

This sounds very strange, so it would be good to start by reviewing log files to gain some context for what you describe.

Most important, we need to see what the error is that occurs when starting.  The logs that would help are:

 

- stderr.log - This will be located in the process directory of the role you are trying to start.  For instance, on the host where my master is running, I might find the log file here:

ls /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep hbase-MASTER| tail -1`/logs

 

- Next, the role log of the role that is not starting.  Normally that will be located in /var/log/hbase

 

You mentioned that the HBase "Service" wasn't starting, but not what HBase Role or Roles were not starting so I couldn't make the log information more specific.

 

Cheers,

 

Ben

Re: HBase fails to start after graceful shutdown until Keytabs are regenerated

Explorer

Hi Ben,

 

Thank you for your prompt response! The roles are as follows: HBase REST server, HBase Thrift Server, Master, 3x RegionServer. 

 

Here are the latest logs I obtained this morning after:

  • stopping (CM - HBase Actions - Stop, I get asked "Are you sure you want to run the HBase Graceful Shutdown command on the service HBase?", I click Stop)
  • and trying to start the HBase service (CM - Hbase Actions - Start, here the prompt asks "Are you sure you want to run the Start command on the service HBase?", I click Start, the process then continues with "Starting 6 roles on service)

... and then it all started without issues. I'm baffled, but also happy that it's working correctly now. I did 2 stop&starts and 1 restart, the issue seems to be gone.

 

Yesterday I was "weeding out" a Java 1.7 installation that popped up on our DEV cluster and it was causing issues with Kerberos. I think it also interfered with our deployment of HBase.

 

Sorry for the false alarm, please feel free to remove this thread.

 

Thank you for your time!

 

Kind regards,

Julius

 

Re: HBase fails to start after graceful shutdown until Keytabs are regenerated

Super Guru

@matagyula,

 

Tough to say, but if it doesn't start happening again, we won't bee too sad.

If it does come back, though, let us know and we can help get to the bottom of it.

Highlighted

Re: HBase fails to start after graceful shutdown until Keytabs are regenerated

Explorer

Hi @matagyula .

 

I think you need to check whether you started zookeeper first or not .

 

i was facing the same issue with same satiation so try to check zookeeper running.

 

then try to start HBase.

 

 

 

 

Thanks

HadoopHelp

 

 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here