Processor PublishKafka can't stop

[#1]

Hello, I stopp a PublishKafka processor on the Web UI. The processor is still running in 2 hosts of the cluster.

I try to find which are the 2 hosts by dragging the connection to the PublishKafka processor to another processor, I get the following message:

It shows me that host bj1233 can not fulfill this request, so I check the log in host bj1233. There are two related lines:

2017-11-15 10:05:23,532 INFO [NiFi Web Server-547070] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10
2017-11-15 10:05:23,532 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PublishKafka_0_10[id=6c643549-32fa-1f10-bad1-7a65935d7466] to run

It makes me confused: the StandardProcessScheduler has stopped the process, but the UI shows that the processor is running.

Did I misunderstand it? How to handle this situation? Any suggestions?
Thanks

[#2]

@fan haiqiang attached is the image that can help.

[#3]

@dhieru singh thanks very much. So it's some kind of feature of PublishKafka 🙂

[#4]

This means there is a thread that is blocked on some operation communicating with Kafka, when this happens please run:

./bin/nifi.sh dump

From the node where the stuck processor is, and then look in nifi-bootstrap.log for the thread dump.

In the thread dump look for a stack that shows something involving Kafka and please provide that portion of the dump.

Restarting the NiFi node where the stuck processor is will resolve the issue, but then we won't know where it was stuck.

[#5]

@Bryan Bende, thanks very much for your replay.
It's a pity that I've already restarted the NiFi node, so I can't get where it was stuck now. When I encounter this problem next time, I'll do what you suggested and post the thread dump here.

Thanks!

[#6]

Hi @Bryan Bende, I encounter this issue again. Unfortunately, I failed to run dump with OOM:

[nifi@sz1314 bin]$ ./nifi.sh dump

Java home: /usr/local/jdk1.8.0_45
NiFi home: /usr/hdf/current/nifi

Bootstrap Config File: /usr/hdf/current/nifi/conf/bootstrap.conf

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at org.apache.nifi.bootstrap.RunNiFi.dump(RunNiFi.java:707)
	at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:232)

Do I need to make a tuning to some setting of NiFi to run dump successfully?
Thanks!

[#7]

@fan haiqiang trying this might help

HADOOP_CLIENT_OPTS="-Xmx2048m" HADOOP_CLIENT_OPTS="-Xmx2048m" before the command you write

Thanks Dheeru

[#8]

@dhieru singh do u mean this

[nifi@sz1314 bin]$ export HADOOP_CLIENT_OPTS="-Xmx2048m"
[nifi@sz1314 bin]$ export HADOOP_CLIENT_OPTS="-Xmx2048m" 
[nifi@sz1314 bin]$ ./nifi.sh dump

[#9]

@fan haiqiang export HADOOP_CLIENT_OPTS="-Xmx2048m" ./nifi.sh dump or try few different ways