Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Could not create hbase table

Could not create hbase table

New Contributor

This following is the exception I'm receiving.

ERROR: java.io.IOException: sync aborted at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2410) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.lang.RuntimeException: sync aborted at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:550) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.insert(WALProcedureStore.java:406) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:752) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil$NonceProcedureRunnable.submitProcedure(MasterProcedureUtil.java:106) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1742) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1731) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:479) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:63400) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352) ... 3 more Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at org.apache.hadoop.util.Shell.runCommand(Shell.java:562) at org.apache.hadoop.util.Shell.run(Shell.java:482) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) at org.apache.hadoop.util.Shell.execCommand(Shell.java:869) at org.apache.hadoop.util.Shell.execCommand(Shell.java:852) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209) at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:465) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:839) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:806) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:741) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:784) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:595) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:67) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:215)

,

This following is the exception I'm receiving.

ERROR: java.io.IOException: sync aborted at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2410) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.lang.RuntimeException: sync aborted at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:550) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.insert(WALProcedureStore.java:406) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:752) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil$NonceProcedureRunnable.submitProcedure(MasterProcedureUtil.java:106) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1742) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1731) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:479) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:63400) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352) ... 3 more Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at org.apache.hadoop.util.Shell.runCommand(Shell.java:562) at org.apache.hadoop.util.Shell.run(Shell.java:482) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) at org.apache.hadoop.util.Shell.execCommand(Shell.java:869) at org.apache.hadoop.util.Shell.execCommand(Shell.java:852) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209) at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:465) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:839) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:806) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:741) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:784) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:595) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:67) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:215)

2 REPLIES 2

Re: Could not create hbase table

Super Collaborator

I guess this could be the root cause of the issue:
3 more Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method)

Based on this log entry, I would check the memory settings for the JVM. It can indicate any kind of out of resource issues within the JVM. First try with

ulimit -a 

if you are running into system limits. Then I can recommend this link, where details are given on how to deal with this issue:

http://javaeesupportpatterns.blogspot.de/2012/09/outofmemoryerror-unable-to-create-new.html

Re: Could not create hbase table

Mentor

@Mohamed Saleem

Could you share the memory steup of your hbase Master and regional servers,belwo is a pointer to that can you increase the memory and restart the hbase and retry

Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at 


What are the current settings of /etc/security/limits.conf file. Below are the new values.

* soft nofile 4096
* hard nofile 128000
hdfs hard nofile 128000
hbase hard nofile 128000
mapred hard nofile 32768
* soft nproc 20000
* hard nproc 65536
hdfs soft/hard nproc 65536
hbase soft/hard nproc 65536
mapred soft/hard nproc 65536

These are my settings.