Skip to main content
hhibigdata
hhibigdata
Silver ⭐️
December 2, 2019
Solved

NiFi ListSTFP hostprovider may not be empty

Hello. I am using nifi cluster with 3 nodes.

I tried to Impala Query with query, it was success.

 

But when I tried to use ListSFTP and FetchSFTP, It occurs Error.

The Error was  

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

 

Why this error occur

 

I changed configuration "nifi.flowcontroller.autoResumeState=false", but still error

 

Can you advise for me?

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
December 3, 2019

@hhibigdata 

 

I am not clear on your setup based on your comments:  I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

 

---  The List based processor are not cluster friendly because the non NiFi protocols they are built for are not cluster friendly.  All this means is that these processors must be configured in your NiFi cluster with an "Execution" of "Primary Node" so that it will only ever be running on one node at a time.  You should not have two different NiFi installs.  In between the ListSFTP and FetchSFTP processors you should be redistributing the listed files via the load balanced strategy options on the connection.

--- NiFi clusters require zookeeper and zookeeper requires quorum meaning you should have an odd number of ZK nodes (3 or 5 recommended).  This same ZK will also be used to store cluster state for thes non cluster friendly processors, so that when a primary node changes nodes, the new node will pull last known state from ZK so that the list based processors continue to list from where previously elected primary node left off.

So two things i suggest you check on:
1. That the zookeeper "Connect String" is correct in your state-management.xml "zk-provider". It should be a comma separated list of 3 to 5 ZK <hostname>:<port>
2. That the "nifi.zookeeper.connect.string=" has been properly setup in the nifi.properties file. It should be a comma separated list of 3 to 5 ZK <hostname>:<port>

*** Generally both use the same ZK connect string and same ZK root node.

 

Hope this helps,

Matt

 

View in thread

5 replies

hhibigdata
hhibigdataAuthor
Silver ⭐️
December 2, 2019

Thanks for you answering.

 

I have a one more question.

 

I configured 3 Nifi Cluster Clustering with 3 node And I tried to use ListSFTP Component in NiFi Cluster, error occurs.

 

The error was

 

ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor.
If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.
Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!
java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)
at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:951)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:688)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.processor.util.list.AbstractListProcessor.updateState(AbstractListProcessor.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142

 

I do not know how to fix it.

In addition to your acticle, you said ListSFTP & ListHDFS were non-cluster friendly.

 

So, When I want to use ListSFTP in NiFi Cluster, I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

 

plz help me..

 

 

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
December 3, 2019

@hhibigdata 

 

I am not clear on your setup based on your comments:  I have to configure one more NiFi Cluster(Standalone)? My NiFi Cluster is 4(3 clustering, 1 standalone)?

 

---  The List based processor are not cluster friendly because the non NiFi protocols they are built for are not cluster friendly.  All this means is that these processors must be configured in your NiFi cluster with an "Execution" of "Primary Node" so that it will only ever be running on one node at a time.  You should not have two different NiFi installs.  In between the ListSFTP and FetchSFTP processors you should be redistributing the listed files via the load balanced strategy options on the connection.

--- NiFi clusters require zookeeper and zookeeper requires quorum meaning you should have an odd number of ZK nodes (3 or 5 recommended).  This same ZK will also be used to store cluster state for thes non cluster friendly processors, so that when a primary node changes nodes, the new node will pull last known state from ZK so that the list based processors continue to list from where previously elected primary node left off.

So two things i suggest you check on:
1. That the zookeeper "Connect String" is correct in your state-management.xml "zk-provider". It should be a comma separated list of 3 to 5 ZK <hostname>:<port>
2. That the "nifi.zookeeper.connect.string=" has been properly setup in the nifi.properties file. It should be a comma separated list of 3 to 5 ZK <hostname>:<port>

*** Generally both use the same ZK connect string and same ZK root node.

 

Hope this helps,

Matt

 

Translate
    Bern
    Bern
    Gold ⭐️
    January 2, 2025

    This solution is not concrete enough, I made the suggested changes for this error "Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the
    Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to
    org.apache.nifi.processor.exception.TerminatedTaskException: org.apache.nifi.processor.exception.TerminatedTaskException", but I still have the problem, modify the state-management.xml, as follows, <cluster-provider>
    <id>zk-provider</id>
    <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
    <property name="Connect String">node-1:2181,node-2:2181,node-3:2181</property>
    <property name="Root Node">/nifi</property>
    <property name="Session Timeout">10 seconds </property>
    <property name="Access Control">Open</property>

    and the zookeeper properties with the following:
    server.1=node-1:2888:3888;2181
    server.2=node-2:2888:3888 ;2181
    server.3=node-3:2888:3888;2181

    I restarted zoo and nifi and the problem is not solved. What's more, I have a slow process to execute. When I execute the flow, the processors take a long time, more than 10 minutes to start. and does not run the generateflowfile processor.error-nifi.JPG

    The processor previously when the cluster worked well the EvaluateJsonPath processor executed in milliseconds now it gets stuck and takes a long time to parse the json where there are some periods of data from to, so that the process works segmented with data, after the select the processors are configured as multitasking, I have nifi 1.11.4 and separately zookeeper 3.6.3

     

    Translate
    stevenmatison
    stevenmatison
    Platinum ⭐️⭐️
    December 2, 2019

    Can you show a screen shot of the processor configuration?

    Translate
      Terms & ConditionsAccessibility statement