Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Meta table is not Online

Highlighted

Meta table is not Online

Champion Alumni

Hi every one,

 

I did some load testing on my hbase cluster based on cdh 4.6 .Data got inserted correctly on hbase tables and also indexed on solr as I  am using lily.How ever  when I try to ingest data now I am getting the  below exception and also not able to scan any tables.

 

Looks like clinet is not able to find meta table from zoo keeper location.Can some one help?

 

 

 

org.apache.hadoop.hbase.client.ScannerCallable@724715dd, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3559)
at org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2557)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2538)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)

10 REPLIES 10
Highlighted

Re: Meta table is not Online

Champion Alumni

Hi ,

 

Can some one help?

Re: Meta table is not Online

Contributor
>From the master web UI, do you see if the meta region in transition?

Highlighted

Re: Meta table is not Online

Champion Alumni

Yes.Meta region is in transition from the master web UI

 

1028785192.META.,,1.1028785192 state=CLOSING
Highlighted

Re: Meta table is not Online

Contributor
How long has it stuck there? You can check where it is trying to open, find
out anything wrong from the log. You may try to manually fail over your
master to see if it can fix the issue. Otherwise, you may need to use hbck.

Highlighted

Re: Meta table is not Online

Champion Alumni

Thank you.It has been in this state for more than a day now.I tried to fail over the master but still the problem persist.

Highlighted

Re: Meta table is not Online

Contributor
That's bad. Do you see anything wrong in the region server log? How many
regions in total do you have?

Highlighted

Re: Meta table is not Online

Champion Alumni

I have 5 region servers in this  trial environment that was set up.There is only one on line region  when I check the master web UI.I ran some heavy load against this yesterday while perfroming a rolling re start  of the cluster.It was all well vbefore that .I am not able to bring it back to work since then.

 

The region server logs does not have any information.

Highlighted

Re: Meta table is not Online

Contributor
Have you tried to restart the whole cluster, use hbck to fix
meta/assignment? Is the log level DEBUG? If so, you should get quite some
information. Without knowing why it is stuck, it would be hard to fix.

Highlighted

Re: Meta table is not Online

New Contributor

Thank you.I tried runing hbase hbck as root user but it says I do not have enough permission.I am  trying run this as user hbase but  It asks  for the habse password.Can you help?

 

 

 

 

14/05/15 16:26:47 WARN util.HBaseFsck: Got AccessControlException when preCheckPermission
org.apache.hadoop.security.AccessControlException: Permission denied: action=WRITE path=hdfs://nameservice1/hbase/-ROOT- user=root
at org.apache.hadoop.hbase.util.FSUtils.checkAccess(FSUtils.java:1333)
at org.apache.hadoop.hbase.util.HBaseFsck.preCheckPermission(HBaseFsck.java:1433)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3653)
at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3502)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3493)

Don't have an account?
Coming from Hortonworks? Activate your account here