Hello, I stopp a PublishKafka processor on the Web UI. The processor is still running in 2 hosts of the cluster.
I try to find which are the 2 hosts by dragging the connection to the PublishKafka processor to another processor, I get the following message:
It shows me that host bj1233 can not fulfill this request, so I check the log in host bj1233. There are two related lines:
2017-11-15 10:05:23,532 INFO [NiFi Web Server-547070] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10 2017-11-15 10:05:23,532 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PublishKafka_0_10[id=6c643549-32fa-1f10-bad1-7a65935d7466] to run
It makes me confused: the StandardProcessScheduler has stopped the process, but the UI shows that the processor is running.
Did I misunderstand it? How to handle this situation? Any suggestions?
This means there is a thread that is blocked on some operation communicating with Kafka, when this happens please run:
From the node where the stuck processor is, and then look in nifi-bootstrap.log for the thread dump.
In the thread dump look for a stack that shows something involving Kafka and please provide that portion of the dump.
Restarting the NiFi node where the stuck processor is will resolve the issue, but then we won't know where it was stuck.
@Bryan Bende, thanks very much for your replay.
It's a pity that I've already restarted the NiFi node, so I can't get where it was stuck now. When I encounter this problem next time, I'll do what you suggested and post the thread dump here.
Hi @Bryan Bende, I encounter this issue again. Unfortunately, I failed to run dump with OOM:
[nifi@sz1314 bin]$ ./nifi.sh dump Java home: /usr/local/jdk1.8.0_45 NiFi home: /usr/hdf/current/nifi Bootstrap Config File: /usr/hdf/current/nifi/conf/bootstrap.conf Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.nifi.bootstrap.RunNiFi.dump(RunNiFi.java:707) at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:232)
Do I need to make a tuning to some setting of NiFi to run dump successfully?