Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Highlighted

Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Expert Contributor

Hi all, I just upgraded from HDP 2.4.2 to HDP 2.5.3 and I am getting this error at this point in the script:

hive --database "archive" -e "msck repair table tablename;"

It used to work fine before upgrade...

2017-02-01 11:25:12,461 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(402)) - Trying to connect to metastore with URI thrift://str21:9083 2017-02-01 11:25:12,513 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(498)) - Connected to metastore. 2017-02-01 11:25:13,710 INFO [main]: session.SessionState (SessionState.java:createPath(673)) - Created local directory: /tmp/6cf6cae3-60ce-4977-8236-a1e38547229d_resources 2017-02-01 11:25:13,720 INFO [main]: session.SessionState (SessionState.java:createPath(673)) - Created HDFS directory: /tmp/hive/v-pgovin/6cf6cae3-60ce-4977-8236-a1e38547229d 2017-02-01 11:25:13,722 INFO [main]: session.SessionState (SessionState.java:createPath(673)) - Created local directory: /tmp/v-pgovin/6cf6cae3-60ce-4977-8236-a1e38547229d 2017-02-01 11:25:13,725 INFO [main]: session.SessionState (SessionState.java:createPath(673)) - Created HDFS directory: /tmp/hive/v-pgovin/6cf6cae3-60ce-4977-8236-a1e38547229d/_tmp_space.db 2017-02-01 11:25:13,746 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:13,746 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:13,777 INFO [main]: ql.Driver (Driver.java:compile(426)) - We are setting the hadoop caller context from to v-pgovin_20170201112513_698062b9-f7f2-42ee-87d9-9546434ad8bb 2017-02-01 11:25:13,782 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:13,786 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: use archive 2017-02-01 11:25:14,778 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed 2017-02-01 11:25:14,780 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=parse start=1485966313782 end=1485966314780 duration=998 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:14,781 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:14,885 INFO [main]: ql.Driver (Driver.java:compile(476)) - Semantic Analysis Completed 2017-02-01 11:25:14,887 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=semanticAnalyze start=1485966314781 end=1485966314887 duration=106 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:14,895 INFO [main]: ql.Driver (Driver.java:getSchema(252)) - Returning Hive schema: Schema(fieldSchemas:null, properties:null) 2017-02-01 11:25:14,896 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile start=1485966313746 end=1485966314896 duration=1150 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:14,896 INFO [main]: ql.Driver (Driver.java:compile(558)) - We are resetting the hadoop caller context to 2017-02-01 11:25:14,896 INFO [main]: ql.Driver (Driver.java:checkConcurrency(172)) - Concurrency mode is disabled, not creating a lock manager 2017-02-01 11:25:14,896 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:14,896 INFO [main]: ql.Driver (Driver.java:execute(1406)) - Setting caller context to query id v-pgovin_20170201112513_698062b9-f7f2-42ee-87d9-9546434ad8bb 2017-02-01 11:25:14,897 INFO [main]: ql.Driver (Driver.java:execute(1409)) - Starting command(queryId=v-pgovin_20170201112513_698062b9-f7f2-42ee-87d9-9546434ad8bb): use archive 2017-02-01 11:25:15,471 INFO [main]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(299)) - Timeline service address: http://str20:8188/ws/v1/timeline/ 2017-02-01 11:25:15,503 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-02-01 11:25:15,504 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,506 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1485966315504 end=1485966315506 duration=2 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,508 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,514 INFO [main]: ql.Driver (Driver.java:launchTask(1743)) - Starting task [Stage-0:DDL] in serial mode 2017-02-01 11:25:15,621 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=runTasks start=1485966315508 end=1485966315621 duration=113 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,621 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-02-01 11:25:15,621 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,623 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1485966315621 end=1485966315623 duration=2 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,623 INFO [main]: ql.Driver (Driver.java:execute(1635)) - Resetting the caller context to 2017-02-01 11:25:15,623 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=Driver.execute start=1485966314896 end=1485966315623 duration=727 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,624 INFO [main]: ql.Driver (SessionState.java:printInfo(980)) - OK 2017-02-01 11:25:15,624 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,624 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1485966315624 end=1485966315624 duration=0 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,624 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=Driver.run start=1485966313746 end=1485966315624 duration=1878 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,626 INFO [main]: CliDriver (SessionState.java:printInfo(980)) - Time taken: 1.882 seconds 2017-02-01 11:25:15,626 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,626 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1485966315626 end=1485966315626 duration=0 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,627 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,627 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,628 INFO [main]: ql.Driver (Driver.java:compile(426)) - We are setting the hadoop caller context from to v-pgovin_20170201112515_bd493371-9b43-4044-b75d-e5143d3bbec4 2017-02-01 11:25:15,628 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,628 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: msck repair table cisco_client_rollup 2017-02-01 11:25:15,630 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed 2017-02-01 11:25:15,630 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=parse start=1485966315628 end=1485966315630 duration=2 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,631 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,753 INFO [main]: ql.Driver (Driver.java:compile(476)) - Semantic Analysis Completed 2017-02-01 11:25:15,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=semanticAnalyze start=1485966315631 end=1485966315753 duration=122 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,754 INFO [main]: ql.Driver (Driver.java:getSchema(252)) - Returning Hive schema: Schema(fieldSchemas:null, properties:null) 2017-02-01 11:25:15,754 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile start=1485966315627 end=1485966315754 duration=127 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,754 INFO [main]: ql.Driver (Driver.java:compile(558)) - We are resetting the hadoop caller context to 2017-02-01 11:25:15,754 INFO [main]: ql.Driver (Driver.java:checkConcurrency(172)) - Concurrency mode is disabled, not creating a lock manager 2017-02-01 11:25:15,754 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,754 INFO [main]: ql.Driver (Driver.java:execute(1406)) - Setting caller context to query id v-pgovin_20170201112515_bd493371-9b43-4044-b75d-e5143d3bbec4 2017-02-01 11:25:15,755 INFO [main]: ql.Driver (Driver.java:execute(1409)) - Starting command(queryId=v-pgovin_20170201112515_bd493371-9b43-4044-b75d-e5143d3bbec4): msck repair table cisco_client_rollup 2017-02-01 11:25:15,755 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-02-01 11:25:15,755 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,756 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1485966315755 end=1485966315756 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,756 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:15,757 INFO [main]: ql.Driver (Driver.java:launchTask(1743)) - Starting task [Stage-0:DDL] in serial mode 2017-02-01 11:25:15,838 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner> 2017-02-01 11:25:16,333 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=partition-retrieving start=1485966315838 end=1485966316333 duration=495 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner> 2017-02-01 11:25:17,728 ERROR [main]: metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread 2017-02-01 11:25:17,747 WARN [main]: exec.DDLTask (DDLTask.java:msck(1828)) - Failed to run metacheck: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(HiveMetaStoreChecker.java:527) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(HiveMetaStoreChecker.java:424) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:315) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:291) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:236) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:113) at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1759) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:378) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:315) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(HiveMetaStoreChecker.java:475) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(HiveMetaStoreChecker.java:62) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:501) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:475) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-02-01 11:25:17,756 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-02-01 11:25:17,756 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:17,757 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1485966317756 end=1485966317757 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:17,757 ERROR [main]: ql.Driver (SessionState.java:printError(989)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 2017-02-01 11:25:17,758 INFO [main]: ql.Driver (Driver.java:execute(1635)) - Resetting the caller context to 2017-02-01 11:25:17,758 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=Driver.execute start=1485966315754 end=1485966317758 duration=2004 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:17,758 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:17,758 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1485966317758 end=1485966317758 duration=0 from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:17,764 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-02-01 11:25:17,764 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1485966317764 end=1485966317764 duration=0 from=org.apache.hadoop.hive.ql.Driver>

7 REPLIES 7

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Rising Star

@PJ

Can you check on Metastore Heap Size if it had changed after upgrade?

Also, check the max user processes for the user running the queries (ulimit -a)

Hope that helps

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Expert Contributor
@icocio

I checked the netastore heap size, it is the same 16GB...I also increased client heap size from 1GB to 4 GB... and Ulimit was only 1024 open files, i cahnged it to 32768 on all nodes....the job is still failing at that point... do i need to restart hdfs ??

Please advise.

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Expert Contributor

I also dis strace and all the threads are saying

Process 5543 attached - interrupt to quit futex(0x7f22b84355c4, FUTEX_WAIT_PRIVATE, 29, NULL^C <unfinished ...>

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Expert Contributor

I checked jstack of hung up process:

/usr/java/jdk1.8.0_101/bin/jstack -F 5273 Attaching to process ID 5273, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.101-b13 Deadlock Detection: No deadlocks found. Thread 5543: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5542: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5541: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5540: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5539: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5538: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5537: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5536: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5535: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5534: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5533: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5532: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5531: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5530: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5529: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.access$100(org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker, java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=10, line=62 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=209, line=501 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call() @bci=1, line=475 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5525: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5519: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - jline.internal.NonBlockingInputStream.run() @bci=44, line=278 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5505: (state = IN_NATIVE) - org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=0 (Interpreted frame) - org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=2, line=52 (Interpreted frame) - org.apache.hadoop.net.unix.DomainSocketWatcher$2.run() @bci=590, line=511 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5504: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame) - org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run() @bci=9, line=3154 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) Thread 5492: (state = BLOCKED) Thread 5491: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame) Thread 5490: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=502 (Compiled frame) - java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Compiled frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Interpreted frame) Thread 5480: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429 (Compiled frame) - java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(java.util.concurrent.ExecutorService, java.util.concurrent.ConcurrentLinkedQueue, java.util.Map, org.apache.hadoop.fs.FileSystem, int, int) @bci=453, line=523 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(org.apache.hadoop.fs.Path, java.util.Set, int) @bci=158, line=424 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(org.apache.hadoop.hive.ql.metadata.Table, java.util.Set, org.apache.hadoop.hive.ql.metadata.CheckResult) @bci=29, line=315 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(org.apache.hadoop.hive.ql.metadata.Table, java.util.List, boolean, org.apache.hadoop.hive.ql.metadata.CheckResult) @bci=220, line=291 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(java.lang.String, java.lang.String, java.util.List, org.apache.hadoop.hive.ql.metadata.CheckResult) @bci=213, line=236 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(java.lang.String, java.lang.String, java.util.List, org.apache.hadoop.hive.ql.metadata.CheckResult) @bci=116, line=113 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.msck(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.MsckDesc) @bci=51, line=1759 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=543, line=378 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=30, line=160 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=7, line=89 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=330, line=1745 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.execute() @bci=895, line=1491 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.runInternal(java.lang.String, boolean) @bci=285, line=1289 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.run(java.lang.String, boolean) @bci=3, line=1156 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=3, line=1146 (Interpreted frame) - org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(java.lang.String, org.apache.hadoop.hive.ql.processors.CommandProcessor, org.apache.hadoop.hive.cli.CliSessionState) @bci=60, line=217 (Interpreted frame) - org.apache.hadoop.hive.cli.CliDriver.processCmd(java.lang.String) @bci=391, line=169 (Interpreted frame) - org.apache.hadoop.hive.cli.CliDriver.processLine(java.lang.String, boolean) @bci=153, line=380 (Interpreted frame) - org.apache.hadoop.hive.cli.CliDriver.executeDriver(org.apache.hadoop.hive.cli.CliSessionState, org.apache.hadoop.hive.conf.HiveConf, org.apache.hadoop.hive.cli.OptionsProcessor) @bci=277, line=740 (Interpreted frame) - org.apache.hadoop.hive.cli.CliDriver.run(java.lang.String[]) @bci=322, line=685 (Interpreted frame) - org.apache.hadoop.hive.cli.CliDriver.main(java.lang.String[]) @bci=8, line=625 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame) - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame) - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame) - org.apache.hadoop.util.RunJar.run(java.lang.String[]) @bci=451, line=233 (Interpreted frame) - org.apache.hadoop.util.RunJar.main(java.lang.String[]) @bci=8, line=148 (Interpreted frame)

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Contributor

Can you plz check with "hive.mv.files.thread=0"

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Contributor

Can you plz attach hive.log and hivemetastore.log?

Re: Hive metadata.HiveMetaStoreChecker (HiveMetaStoreChecker.java:checkPartitionDirs(525)) - java.lang.OutOfMemoryError: unable to create new native thread

Expert Contributor

AFTER set hive.mv.files.thread=0 is set, the error is:

2017-02-02 14:43:08,832 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,833 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,833 INFO [main]: ql.Driver (Driver.java:compile(426)) - We are setting the hadoop caller context from to hive_20170202144308_386f5b23-6a97-4b1c-ad7d-def91305d0b1 2017-02-02 14:43:08,834 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,834 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: msck repair table cisco_client_rollup 2017-02-02 14:43:08,836 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed 2017-02-02 14:43:08,836 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=parse start=1486064588834 end=1486064588836 duration=2 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,836 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,878 INFO [main]: ql.Driver (Driver.java:compile(476)) - Semantic Analysis Completed 2017-02-02 14:43:08,878 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=semanticAnalyze start=1486064588836 end=1486064588878 duration=42 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,878 INFO [main]: ql.Driver (Driver.java:getSchema(252)) - Returning Hive schema: Schema(fieldSchemas:null, properties:null) 2017-02-02 14:43:08,878 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile start=1486064588833 end=1486064588878 duration=45 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,879 INFO [main]: ql.Driver (Driver.java:compile(558)) - We are resetting the hadoop caller context to 2017-02-02 14:43:08,879 INFO [main]: ql.Driver (Driver.java:checkConcurrency(172)) - Concurrency mode is disabled, not creating a lock manager 2017-02-02 14:43:08,879 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,879 INFO [main]: ql.Driver (Driver.java:execute(1406)) - Setting caller context to query id hive_20170202144308_386f5b23-6a97-4b1c-ad7d-def91305d0b1 2017-02-02 14:43:08,879 INFO [main]: ql.Driver (Driver.java:execute(1409)) - Starting command(queryId=hive_20170202144308_386f5b23-6a97-4b1c-ad7d-def91305d0b1): msck repair table cisco_client_rollup 2017-02-02 14:43:08,880 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-02-02 14:43:08,880 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,881 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1486064588880 end=1486064588881 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,881 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:08,882 INFO [main]: ql.Driver (Driver.java:launchTask(1743)) - Starting task [Stage-0:DDL] in serial mode 2017-02-02 14:43:08,964 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner> 2017-02-02 14:43:09,499 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=partition-retrieving start=1486064588964 end=1486064589498 duration=534 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner> 2017-02-02 14:43:38,782 ERROR [main]: exec.DDLTask (DDLTask.java:failed(533)) - java.lang.NullPointerException at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(HiveMetaStoreChecker.java:425) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:315) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:291) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:236) at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:113) at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1759) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:378) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:740) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) 2017-02-02 14:43:38,782 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-02-02 14:43:38,782 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:38,784 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1486064618782 end=1486064618784 duration=2 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:38,784 ERROR [main]: ql.Driver (SessionState.java:printError(989)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. null 2017-02-02 14:43:38,784 INFO [main]: ql.Driver (Driver.java:execute(1635)) - Resetting the caller context to 2017-02-02 14:43:38,784 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=Driver.execute start=1486064588879 end=1486064618784 duration=29905 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:38,785 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:38,785 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1486064618785 end=1486064618785 duration=0 from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:38,791 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-02-02 14:43:38,791 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1486064618791 end=1486064618791 duration=0 from=org.apache.hadoop.hive.ql.Driver>

Don't have an account?
Coming from Hortonworks? Activate your account here