Support Questions
Find answers, ask questions, and share your expertise

putHDFS thread gets hung up

putHDFS thread gets hung up

Contributor

Hi All,

Thanks a lot this awesome community.

I am writing to HDFS using PutHDFS and every now and then it gets hung up. Attached is the thread dump

"Timer-Driven Process Thread-29" Id=178 BLOCKED  on java.io.BufferedInputStream@2f8361ea
	at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	- waiting on java.io.InputStreamReader@b94e0ab
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	- waiting on java.io.InputStreamReader@b94e0ab
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
	at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
	at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
	at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
	at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
	at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:560)
	- waiting on org.apache.hadoop.ipc.Client$Connection@6edf66a8
	at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:729)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
	- waiting on org.apache.hadoop.ipc.Client$Connection@6edf66a8
	at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
	at org.apache.hadoop.ipc.Client.call(Client.java:1451)
	at org.apache.hadoop.ipc.Client.call(Client.java:1412)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy459.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
	at sun.reflect.GeneratedMethodAccessor361.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy460.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
	at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
	at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:223)
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Number of Locked Synchronizers: 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@73dd1f4d

Is this because of some race condition?

any help or suggestion

Thanks

Dheeru

3 REPLIES 3

Re: putHDFS thread gets hung up

Rising Star
@dhieru singh

The issue your running into is probably the same thing that @Tarun Kumar is experiencing. Please see my reply to that question [1] and follow the steps outlined to provide some more information on what's happening with your cluster. I am happy take a look at the information you provide.

[1] https://community.hortonworks.com/questions/155101/nifi-puthdfs-processor-stuck-and-not-able-to-rele...

Re: putHDFS thread gets hung up

Contributor

@Jeff Storck sure I will be happy to provide you the details, it may take some time because of the holiday season Thanks

Dhieru

Re: putHDFS thread gets hung up

Rising Star
@dhieru singh

No worries. I wanted to let you know that we are actively working this issue and any debug logging or details you can provide to consistently reproduce the issue (NiFi settings, environment settings, JAAS configs, etc) will be helpful.