Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi ListSTFP hostprovider may not be empty

Solved Go to solution
Highlighted

NiFi ListSTFP hostprovider may not be empty

New Contributor

Hello. I am using nifi cluster with 3 nodes.

I tried to Impala Query with query, it was success.

 

But when I tried to use ListSFTP and FetchSFTP, It occurs Error.

The Error was  

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

 

Why this error occur

 

I changed configuration "nifi.flowcontroller.autoResumeState=false", but still error

 

Can you advise for me?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: NiFi ListSTFP

Master Guru

@hhibigdata 

 

I am not clear on your setup based on your comments:  I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

 

---  The List based processor are not cluster friendly because the non NiFi protocols they are built for are not cluster friendly.  All this means is that these processors must be configured in your NiFi cluster with an "Execution" of "Primary Node" so that it will only ever be running on one node at a time.  You should not have two different NiFi installs.  In between the ListSFTP and FetchSFTP processors you should be redistributing the listed files via the load balanced strategy options on the connection.

--- NiFi clusters require zookeeper and zookeeper requires quorum meaning you should have an odd number of ZK nodes (3 or 5 recommended).  This same ZK will also be used to store cluster state for thes non cluster friendly processors, so that when a primary node changes nodes, the new node will pull last known state from ZK so that the list based processors continue to list from where previously elected primary node left off.

So two things i suggest you check on:
1. That the zookeeper "Connect String" is correct in your state-management.xml "zk-provider". It should be a comma separated list of 3 to 5 ZK <hostname>:<port>
2. That the "nifi.zookeeper.connect.string=" has been properly setup in the nifi.properties file. It should be a comma separated list of 3 to 5 ZK <hostname>:<port>

*** Generally both use the same ZK connect string and same ZK root node.

 

Hope this helps,

Matt

 

3 REPLIES 3

Re: NiFi ListSTFP hostprovider may not be empty

Super Collaborator

Can you show a screen shot of the processor configuration?

Re: NiFi ListSTFP

New Contributor

Thanks for you answering.

 

I have a one more question.

 

I configured 3 Nifi Cluster Clustering with 3 node And I tried to use ListSFTP Component in NiFi Cluster, error occurs.

 

The error was

 

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor.
If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.
Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

 

I do not know how to fix it.

In addition to your acticle, you said ListSFTP & ListHDFS were non-cluster friendly.

 

So, When I want to use ListSFTP in NiFi Cluster, I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

 

plz help me..

 

 

Re: NiFi ListSTFP

Master Guru

@hhibigdata 

 

I am not clear on your setup based on your comments:  I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

 

---  The List based processor are not cluster friendly because the non NiFi protocols they are built for are not cluster friendly.  All this means is that these processors must be configured in your NiFi cluster with an "Execution" of "Primary Node" so that it will only ever be running on one node at a time.  You should not have two different NiFi installs.  In between the ListSFTP and FetchSFTP processors you should be redistributing the listed files via the load balanced strategy options on the connection.

--- NiFi clusters require zookeeper and zookeeper requires quorum meaning you should have an odd number of ZK nodes (3 or 5 recommended).  This same ZK will also be used to store cluster state for thes non cluster friendly processors, so that when a primary node changes nodes, the new node will pull last known state from ZK so that the list based processors continue to list from where previously elected primary node left off.

So two things i suggest you check on:
1. That the zookeeper "Connect String" is correct in your state-management.xml "zk-provider". It should be a comma separated list of 3 to 5 ZK <hostname>:<port>
2. That the "nifi.zookeeper.connect.string=" has been properly setup in the nifi.properties file. It should be a comma separated list of 3 to 5 ZK <hostname>:<port>

*** Generally both use the same ZK connect string and same ZK root node.

 

Hope this helps,

Matt

 

Don't have an account?
Coming from Hortonworks? Activate your account here