NiFi ListSTFP hostprovider may not be empty

[#1]

Hello. I am using nifi cluster with 3 nodes.

I tried to Impala Query with query, it was success.

But when I tried to use ListSFTP and FetchSFTP, It occurs Error.

The Error was  

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

Why this error occur

I changed configuration "nifi.flowcontroller.autoResumeState=false", but still error

Can you advise for me?

[#2]

Can you show a screen shot of the processor configuration?

[#3]

Thanks for you answering.

I have a one more question.

I configured 3 Nifi Cluster Clustering with 3 node And I tried to use ListSFTP Component in NiFi Cluster, error occurs.

The error was

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor.
If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.
Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

I do not know how to fix it.

In addition to your acticle, you said ListSFTP & ListHDFS were non-cluster friendly.

So, When I want to use ListSFTP in NiFi Cluster, I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

plz help me..

[#4 · Accepted solution]

@hhibigdata 

I am not clear on your setup based on your comments:  I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

---  The List based processor are not cluster friendly because the non NiFi protocols they are built for are not cluster friendly.  All this means is that these processors must be configured in your NiFi cluster with an "Execution" of "Primary Node" so that it will only ever be running on one node at a time.  You should not have two different NiFi installs.  In between the ListSFTP and FetchSFTP processors you should be redistributing the listed files via the load balanced strategy options on the connection.

--- NiFi clusters require zookeeper and zookeeper requires quorum meaning you should have an odd number of ZK nodes (3 or 5 recommended).  This same ZK will also be used to store cluster state for thes non cluster friendly processors, so that when a primary node changes nodes, the new node will pull last known state from ZK so that the list based processors continue to list from where previously elected primary node left off.

So two things i suggest you check on:
1. That the zookeeper "Connect String" is correct in your state-management.xml "zk-provider". It should be a comma separated list of 3 to 5 ZK <hostname>:<port>
2. That the "nifi.zookeeper.connect.string=" has been properly setup in the nifi.properties file. It should be a comma separated list of 3 to 5 ZK <hostname>:<port>

*** Generally both use the same ZK connect string and same ZK root node.

Hope this helps,

Matt