Reply
Highlighted
Explorer
Posts: 14
Registered: ‎06-06-2017

Phoenix Hbase Crash Issue

Hi Guys.

recently, i met a phoenix + hbase issues.

I have a phoenix table with index. When large data coming to phoenix will case the hbase server crashed.

Already increase the heap size to 32GB

anyone meet same case:

 

2017-10-12 02:55:03,434 INFO org.apache.hadoop.hbase.regionserver.HRegion: Flushing 1/1 column families, memstore=32.05 MB; WAL is null, using passed sequenceid=96594
2017-10-12 02:55:03,478 INFO org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-3414533165141223590 region=SAP_PARTICIPANTDATA,\x0A,1501812516178.838d48dd13880fb2ef6478639c5e1195. Index update failed
2017-10-12 02:55:03,509 WARN org.apache.hadoop.hbase.client.AsyncProcess: Caught unexpected exception/error:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1028)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$100(AsyncProcess.java:615)
at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:597)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:974)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:991)
at org.apache.hadoop.hbase.client.HTableWrapper.batch(HTableWrapper.java:255)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:171)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:129)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Announcements