question Re: Work putHDFS on HDFS in HA? in Archives of Support Questions (Read Only)

Work putHDFS on HDFS in HA?

isoardi — Wed, 03 Feb 2016 23:56:23 GMT

I have a NiFi job but putHDFS processor return an error:

failed to invoke @OnScheduled method due to java.lang.IllegalArgumentException: java.net.UnknownHostException: provaha; processor will not be scheduled to run for 30 sec: java.lang.IllegalArgumentException: java.net.UnknownHostException: provaha

provaha is the correct reference for HDFS High Availability.

is there any particular configuration for puthdfs that it work with HDFS in HA?

Re: Work putHDFS on HDFS in HA?

aervits — Wed, 03 Feb 2016 23:58:28 GMT

did you try passing /etc/hadoop/conf/core-site.xml to putHDFS processor @Davide Isoardi

Re: Work putHDFS on HDFS in HA?

isoardi — Mon, 19 Aug 2019 10:16:22 GMT

Yes @Artem Ervits, I passing the correct path to core-site.xlm, but putHDFS don't work.

Re: Work putHDFS on HDFS in HA?

aervits — Thu, 04 Feb 2016 00:04:19 GMT

@jwitt @mpayne

Re: Work putHDFS on HDFS in HA?

mpayne — Thu, 04 Feb 2016 00:10:01 GMT

It looks like it cannot resolve the hostname 'provaha' to an IP address - can you check if the provaha is in your /etc/hosts file, or supply the FQDN instead?

Re: Work putHDFS on HDFS in HA?

mburgess — Thu, 04 Feb 2016 02:43:25 GMT

That probably works but defeats the purpose of HA. Hadoop is supposed to dynamically resolve 'provaha' to any of the physical nodes in the HA cluster. I wonder if PutHDFS is not successfully getting the physical node resolved from the HA node name, or if it is an error in config (core-site.xml)

Re: Work putHDFS on HDFS in HA?

mmiklavcic — Thu, 04 Feb 2016 04:37:39 GMT

You're missing hdfs-site.xml in the config, which is where the NN HA details are found. Config requires both hdfs-site and core-site,

i.e., set "Hadoop Configuration Resources" similar to the following:

/etc/hadoop/2.3.4.0-3485/0/hdfs-site.xml,/etc/hadoop/2.3.4.0-3485/0/core-site.xml

Reference:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html

A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.

Re: Work putHDFS on HDFS in HA?

maher1 — Thu, 16 Jun 2016 20:39:35 GMT

Please provide full path with reference to root directory (absolute path). ?/?/?/etc/hadoop/conf/core-site.xml then it should work fine.

Re: Work putHDFS on HDFS in HA?

TimothySpann — Fri, 17 Jun 2016 02:02:36 GMT

core-site has worked for me