Support Questions
Find answers, ask questions, and share your expertise

Insert overwrite directory query makes hiveserver2 hanging in HDP-2.3.2.0-2950

New Contributor

Hi,

I encountered a very wired situation. If I try to execute "insert overwrite directory 'pathtosavefileinhdfs' select * from table", Hiveserver2 makes a lot of threads for connecting namenode (url:8020) then it hangs forever.

The thread dump is as below :

..omited..

Locked ownable synchronizers: - locked <4f32f430> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3820
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <2e6e868e> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"RMI TCP Connection(7)-0.0.0.0" - Thread t@3821
  java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   at java.net.SocketInputStream.read(SocketInputStream.java:170)
   at java.net.SocketInputStream.read(SocketInputStream.java:141)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
   - locked <4388576f> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(FilterInputStream.java:83)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$8/1059608701.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Native Method)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
  Locked ownable synchronizers:
   - locked <a307b12> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3819
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <3ed2aab0> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3818
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <5024aaa3> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3817
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <5f32ec2b> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3816
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <37e8dc80> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3815
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <30d2147a> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3814
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <72cd920a> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - Thread t@3813
  java.lang.Thread.State: TIMED_WAITING
   at java.lang.Object.wait(Native Method)
   - waiting on <64550aaf> (a org.apache.hadoop.ipc.Client$Connection)
   at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  Locked ownable synchronizers:
   - None
"IPC Client (657252457) connection to hostname/0.0.0.0:8020 from ldapid" - 

Is there any solution for that?

Thanks,

Kwangwoo

2 REPLIES 2

@Kwangwoo Kim could you please share complete jstack?

Contributor

IPC Client threads are harmless. You can change "ipc.client.connection.maxidletime". Default is 10000 which is 10 seconds, after which it would shutdown if its idle.