Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Insert overwrite directory query makes hiveserver2 hanging in HDP-2.3.2.0-2950

Highlighted

Insert overwrite directory query makes hiveserver2 hanging in HDP-2.3.2.0-2950

New Contributor

Hi,

I encountered a very wired situation. If I try to execute "insert overwrite directory 'pathtosavefileinhdfs' select * from table", Hiveserver2 makes a lot of threads for connecting namenode (url:8020) then it hangs forever.

The thread dump is as below :

..omited..

Locked ownable synchronizers: - locked <4f32f430> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3820
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <2e6e868e> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"RMI TCP Connection(7)-0.0.0.0" - Thread t@3821
  java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   at java.net.SocketInputStream.read(SocketInputStream.java:170)
   at java.net.SocketInputStream.read(SocketInputStream.java:141)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
   - locked <4388576f> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(FilterInputStream.java:83)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$8/1059608701.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Native Method)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
  Locked ownable synchronizers:
   - locked <a307b12> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3819
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <3ed2aab0> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3818
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <5024aaa3> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3817
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <5f32ec2b> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3816
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <37e8dc80> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3815
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <30d2147a> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3814
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <72cd920a> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3813
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <64550aaf> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - 

Is there any solution for that?

Thanks,

Kwangwoo

2 REPLIES 2
Highlighted

Re: Insert overwrite directory query makes hiveserver2 hanging in HDP-2.3.2.0-2950

@Kwangwoo Kim could you please share complete jstack?

Highlighted

Re: Insert overwrite directory query makes hiveserver2 hanging in HDP-2.3.2.0-2950

Contributor

IPC Client threads are harmless. You can change "ipc.client.connection.maxidletime". Default is 10000 which is 10 seconds, after which it would shutdown if its idle.

Don't have an account?
Coming from Hortonworks? Activate your account here