Support Questions

Find answers, ask questions, and share your expertise

Problems connecting to hdfs using Nifi-1.8

Upgraded Nifi from 1.6 to 1.8 and now encountering the following error when trying to connect to HDFS.

o.apache.nifi.processors.hadoop.GetHDFS GetHDFS[id=a9260432-0167-1000-dc86-f7da4cc66fda] Error while retrieving list of files due to java.net.ConnectException: Call From xxxxxxxx.xxxxxxxx.com/xxx.xxx.xxx.xxx to xxx.xxxxxxxxx.com:9820 failed on connection exception: java.net.ConnectException: Connection refused

Some more detail

We have 4 instances of Nifi and all of them were upgraded. Two of them work fine, and two do not. The problem appears to be environmental as the two instances that do not work are hosted at a different provider than the two that do work.

I've tracked the issue to the nifi-hadoop-libraries-nar-1.8.0.nar file. When that file is used, we get the error above. When we replace that file by linking to the old nifi-hadoop-libraries-nar-1.6.0.nar file, we can talk to HDFS just fine.

We are on CDH 5.9.3 on all four servers.

I'm curious to know if anyone else has seen this problem and if anyone has suggestions on where to look to troubleshoot the problem.

Thanks

5 REPLIES 5

Explorer

Hi @Alex Parkhurst,

just to clarify, the 2 instances that work fine are same os/nifi/hdfs client version as the 2 that seem to be not working?

Yes, same everything.

Explorer

The only thing that pops into my mind is that 2 server are still running 1.6 NiFi while trying to access 1.8 resources.

bat as far as i know NiFi should not allow you to run mixed version in the same cluster .

could you post the error that you get while using the 1.8 resource instead of the 1.6?

This is the error reported in the logs and on the Nifi processor in the UI.

o.apache.nifi.processors.hadoop.GetHDFS GetHDFS[id=a9260432-0167-1000-dc86-f7da4cc66fda] Error while retrieving list of files due to java.net.ConnectException: Call From xxxxxxxx.xxxxxxxx.com/xxx.xxx.xxx.xxx to xxx.xxxxxxxxx.com:9820 failed on connection exception: java.net.ConnectException: Connection refused

I'm not a Unix admin, just good enough to be dangerous, but my suspicion is a java version problem. I posted here hoping someone had seen it before and could give me a pointer, but looks like no. I'll keep poking around.

Thanks.

Problem turned out to be the correct port was not 9820, but rather 8020. Adding that port to the fs.defaultFS property in core-site.xml resolved the issue. Something I had tried to verify earlier in troubleshooting but for a variety of reasons getting the config info from the Data Center management team was challenging. Thanks for the help.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.