Phoenix issue RuntimeException: java.lang.OutOfMemoryError Unable to create new native thread

amol_08 · ‎11-15-2018

I am getting below error when I am running the query :

      at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
        at org.apache.phoenix.schema.stats.StatisticsUtil.readStatistics(StatisticsUtil.java:160)
        at org.apache.phoenix.query.TableStatsCache$StatsLoader.load(TableStatsCache.java:92)
        at org.apache.phoenix.query.TableStatsCache$StatsLoader.load(TableStatsCache.java:83)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
        ... 21 more
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
        at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
        at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1259)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
        ... 33 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)

asinghal · ‎11-15-2018

you may need to check your operating system limits i.e no. of threads allowed to run for a user, you can use "ulimit –u" or cat /proc/sys/kernel/threads-max on the shell.

Try increasing the limit

echo 100000 > /proc/sys/kernel/threads-max