Created on 11-15-2017 03:39 AM - edited 08-17-2019 11:16 PM
Hello, I stopp a PublishKafka processor on the Web UI. The processor is still running in 2 hosts of the cluster.
I try to find which are the 2 hosts by dragging the connection to the PublishKafka processor to another processor, I get the following message:
It shows me that host bj1233 can not fulfill this request, so I check the log in host bj1233. There are two related lines:
2017-11-15 10:05:23,532 INFO [NiFi Web Server-547070] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10 2017-11-15 10:05:23,532 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PublishKafka_0_10[id=6c643549-32fa-1f10-bad1-7a65935d7466] to run
It makes me confused: the StandardProcessScheduler has stopped the process, but the UI shows that the processor is running.
Did I misunderstand it? How to handle this situation? Any suggestions?
Created on 11-16-2017 05:31 PM - edited 08-17-2019 11:16 PM
Created 12-15-2017 02:27 PM
Created 11-16-2017 08:49 PM
This means there is a thread that is blocked on some operation communicating with Kafka, when this happens please run:
From the node where the stuck processor is, and then look in nifi-bootstrap.log for the thread dump.
In the thread dump look for a stack that shows something involving Kafka and please provide that portion of the dump.
Restarting the NiFi node where the stuck processor is will resolve the issue, but then we won't know where it was stuck.
Created 11-17-2017 01:46 PM
@Bryan Bende, thanks very much for your replay.
It's a pity that I've already restarted the NiFi node, so I can't get where it was stuck now. When I encounter this problem next time, I'll do what you suggested and post the thread dump here.
Created 12-03-2017 03:57 PM
Hi @Bryan Bende, I encounter this issue again. Unfortunately, I failed to run dump with OOM:
[nifi@sz1314 bin]$ ./nifi.sh dump Java home: /usr/local/jdk1.8.0_45 NiFi home: /usr/hdf/current/nifi Bootstrap Config File: /usr/hdf/current/nifi/conf/bootstrap.conf Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.nifi.bootstrap.RunNiFi.dump(RunNiFi.java:707) at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:232)
Do I need to make a tuning to some setting of NiFi to run dump successfully?
Created 12-04-2017 10:37 PM
@fan haiqiang trying this might help
HADOOP_CLIENT_OPTS="-Xmx2048m" HADOOP_CLIENT_OPTS="-Xmx2048m" before the command you write
Created 12-15-2017 02:30 PM
@dhieru singh do u mean this
[nifi@sz1314 bin]$ export HADOOP_CLIENT_OPTS="-Xmx2048m" [nifi@sz1314 bin]$ export HADOOP_CLIENT_OPTS="-Xmx2048m" [nifi@sz1314 bin]$ ./nifi.sh dump
Created 12-15-2017 06:21 PM