question Re: NiFi ListSTFP in Support Questions

NiFi ListSTFP hostprovider may not be empty

hhibigdata — Mon, 02 Dec 2019 09:32:59 GMT

Hello. I am using nifi cluster with 3 nodes.

I tried to Impala Query with query, it was success.

But when I tried to use ListSFTP and FetchSFTP, It occurs Error.

The Error was

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

Why this error occur

I changed configuration "nifi.flowcontroller.autoResumeState=false", but still error

Can you advise for me?

Re: NiFi ListSTFP

hhibigdata — Mon, 02 Dec 2019 21:45:11 GMT

Thanks for you answering.

I have a one more question.

I configured 3 Nifi Cluster Clustering with 3 node And I tried to use ListSFTP Component in NiFi Cluster, error occurs.

The error was

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor.
If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.
Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

I do not know how to fix it.

In addition to your acticle, you said ListSFTP & ListHDFS were non-cluster friendly.

So, When I want to use ListSFTP in NiFi Cluster, I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

plz help me..

Re: NiFi ListSTFP hostprovider may not be empty

stevenmatison — Mon, 02 Dec 2019 21:27:28 GMT

Can you show a screen shot of the processor configuration?

Re: NiFi ListSTFP

MattWho — Tue, 03 Dec 2019 20:06:57 GMT

@hhibigdata

I am not clear on your setup based on your comments: I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

--- The List based processor are not cluster friendly because the non NiFi protocols they are built for are not cluster friendly. All this means is that these processors must be configured in your NiFi cluster with an "Execution" of "Primary Node" so that it will only ever be running on one node at a time. You should not have two different NiFi installs. In between the ListSFTP and FetchSFTP processors you should be redistributing the listed files via the load balanced strategy options on the connection.

--- NiFi clusters require zookeeper and zookeeper requires quorum meaning you should have an odd number of ZK nodes (3 or 5 recommended). This same ZK will also be used to store cluster state for thes non cluster friendly processors, so that when a primary node changes nodes, the new node will pull last known state from ZK so that the list based processors continue to list from where previously elected primary node left off.

So two things i suggest you check on:
1. That the zookeeper "Connect String" is correct in your state-management.xml "zk-provider". It should be a comma separated list of 3 to 5 ZK <hostname>:<port>
2. That the "nifi.zookeeper.connect.string=" has been properly setup in the nifi.properties file. It should be a comma separated list of 3 to 5 ZK <hostname>:<port>

*** Generally both use the same ZK connect string and same ZK root node.

Hope this helps,

Matt

Re: NiFi ListSTFP

Bern — Thu, 02 Jan 2025 21:59:48 GMT

This solution is not concrete enough, I made the suggested changes for this error "Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the
Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to
org.apache.nifi.processor.exception.TerminatedTaskException: org.apache.nifi.processor.exception.TerminatedTaskException", but I still have the problem, modify the state-management.xml, as follows, <cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Connect String">node-1:2181,node-2:2181,node-3:2181</property>
<property name="Root Node">/nifi</property>
<property name="Session Timeout">10 seconds </property>
<property name="Access Control">Open</property>

and the zookeeper properties with the following:
server.1=node-1:2888:3888;2181
server.2=node-2:2888:3888 ;2181
server.3=node-3:2888:3888;2181

I restarted zoo and nifi and the problem is not solved. What's more, I have a slow process to execute. When I execute the flow, the processors take a long time, more than 10 minutes to start. and does not run the generateflowfile processor.

The processor previously when the cluster worked well the EvaluateJsonPath processor executed in milliseconds now it gets stuck and takes a long time to parse the json where there are some periods of data from to, so that the process works segmented with data, after the select the processors are configured as multitasking, I have nifi 1.11.4 and separately zookeeper 3.6.3

Re: NiFi ListSTFP

MattWho — Tue, 07 Jan 2025 15:17:32 GMT

@Bern

I suggest starting a new community question with the full error stack trace you are seeing. Your exception seems different then the one discussed in this community question:

Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!

You exception is:

Failure is due to org.apache.nifi.processor.exception.TerminatedTaskException:

A few observations and things you may want to provide details around in your new community post:
1. The version of Apache NiFi you are using was released ~6 years ago. You should really consider upgrading to take advantage of lost of bug fixes, performance improvements, new features, and security CVEs addressed. The latest release in the 1.x branch is 1.28 (which is final release of 1.x branch).

2. Your screenshot shows over 250,000 queued FlowFiles (25.75 GB) and 1.373 running processors components. What do you have set as your Max Rimer Driven Tread count?

3. Any other WARN or ERROR messages in your NiFi logs? Any Out of Memory (OOM) reported?

4. It does not make sense why you are load-balancing in so many connections?

Thank you,
Matt