Support Questions
Find answers, ask questions, and share your expertise

'ListHDFS' processor is not working.

'ListHDFS' processor is not working.

Rising Star

Hello,

I am trying to use 'ListHDFS' processor and it is giving error and not listing the files.

I have configured DistributedMapCacheServer & Client and enabled it.

Please find the below error screen shot -


And when I try to right click on the processor and 'view state' it is giving below error :


Thanks in advance !

Mahendra

5 REPLIES 5

Re: 'ListHDFS' processor is not working.

Rising Star
@Mahendra Hegde

It looks like your ZooKeeper server(s) may be down. Can you check the status of that service in Ambari?

Also, if you can find the point where these errors are occurring in nifi-app.log on the NiFi node on which this is happening, there may be more information. Can you attach that log here so I can take a look at it?

Re: 'ListHDFS' processor is not working.

Explorer

I got the same issue, and in my nifi-app.log, I have the following message: ( I have 3 nodes and set the Server Hostname for DistributedMapCacheClientService, I used myserver.dev.test.com. I tried localhost, and got the same error too.) Any suggestion is appreciated. Thanks a lot.

2018-03-02 16:04:52,177 ERROR [StandardProcessScheduler Thread-8] o.a.n.p.standard.MonitorActivity MonitorActivity[id=ab5f82a3-ba56-3764-abf5-5d665d0049fb] Failed to clear cluster state due to java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb java.io.IOException:

Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:347) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453) at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.clear(StandardStateManagerProvider.java:292) at org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85) at org.apache.nifi.processors.standard.MonitorActivity.onStopped(MonitorActivity.java:392) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:233) at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:85) at org.apache.nifi.controller.StandardProcessorNode$2.run(StandardProcessorNode.java:1404) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Caused by: java.net.UnknownHostException: myserver.dev.test.com at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61) at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:316) ... 22 common frames omitted

Re: 'ListHDFS' processor is not working.

@Mahendra Hegde

Shutdown NiFi services

Restart zookeepers

Start NiFi services

If the error is still there, post last errors from nifi and zookeeper logs. Most likely one of zookeepers failed. Zookeeper log can reveal the cause.

Re: 'ListHDFS' processor is not working.

Explorer

@Constantin Stanca @Jeff Storck

I had created a 2 nodes clusters with 4 zookeepers from scratch on 2 VM (1 nifi and 2 zookeepers on each VM), but I still get the same errors. I also setup DistributedMapCacheClientService & DistributedMapCacheServe.

I attached the logs (nifi and zookeeprs). Any suggestion will be appreciated. Thanks a lot.

Andy

Here is the setttings
1)

server 1 zookeeper-3.4.11/zookeeper-1/bin/ (Follower)

zookeeper-3.4.11/zookeeper-2/bin/ (Follower)

nifi-1.4.0/bin (primary, Coordinator)

2) server 2 zookeeper-3.4.11/zookeeper-3/bin/ (Leader)

zookeeper-3.4.11/zookeeper-4/bin/ (Follower)

nifi-1.4.0/bin

3) zookeeper logs lsof -i:2181 |grep LISTEN lsof -p 1649 | grep log lsof -p 12093 | grep out $ lsof -i:2181 |grep LISTEN java 12041 pdn0app1 25u IPv4 356369590 0t0 TCP *:eforward (LISTEN) $ lsof -p 12041 | grep log java 12041 pdn0app1 mem REG 253,5 481535 54658050 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/log4j-1.2.16.jar java 12041 pdn0app1 mem REG 253,5 9753 54658055 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/slf4j-log4j12-1.6.1.jar java 12041 pdn0app1 5r REG 253,5 9753 54658055 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/slf4j-log4j12-1.6.1.jar java 12041 pdn0app1 8r REG 253,5 481535 54658050 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/log4j-1.2.16.jar java 12041 pdn0app1 34w REG 253,5 67108880 54791040 /app/DN00/zookeeper-3.4.11/zookeeper-3/data/version-2/log.200000001 $ lsof -p 12041 | grep out java 12041 pdn0app1 1w REG 253,5 14225 54659929 /app/DN00/zookeeper-3.4.11/zookeeper-3/bin/zookeeper.out java 12041 pdn0app1 2w REG 253,5 14225 54659929 /app/DN00/zookeeper-3.4.11/zookeeper-3/bin/zookeeper.out

Re: 'ListHDFS' processor is not working.

@Mahendra Hegde

Zookeepers need to be odd number to allow electing a lead, e.g. 1, 3, 5 ...