Sqoop UnknownHostException when using HA namenode

owen_matthews — Mon, 18 Jul 2016 21:34:59 GMT

We're getting an UnknownHostException when running Sqoop.

The job gets submitted successfully to the cluster, but then fails almost immediately. In the job history logs, we can see the exception stack trace below.

Background details:

client OS is RHEL 6.7
Sqoop version is 1.4.6.2.3.2.0-2950
Our cluster is running HDP 2.4.2
Our cluster is configured with an HA namenode, called 'titan'.
We're using Kerberos for authentication.
We can run other MapReduce and Spark jobs successfully, it is just Sqoop that is failing with this error.
We can run this Sqoop job on an older (HDP 2.3.2) cluster successfully (it also uses HA namenode).

Stacktrace:

2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false 2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false 2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false 2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.domain.socket.path = 2016-07-15 16:51:34,807 DEBUG [main] org.apache.hadoop.service.AbstractService: noteFailure org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan 2016-07-15 16:51:34,807 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:519) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:499) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1598) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:499) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:285) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1556) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486) Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:66) at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:50) at org.kitesdk.data.spi.Registration.lookupDatasetUri(Registration.java:106) at org.kitesdk.data.Datasets.load(Datasets.java:103) at org.kitesdk.data.Datasets.load(Datasets.java:165) at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510) at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:517) ... 11 more Caused by: java.net.UnknownHostException: titan ... 30 more 2016-07-15 16:51:34,808 DEBUG [main] org.apache.hadoop.service.AbstractService: Service: org.apache.hadoop.mapreduce.v2.app.MRAppMaster entered state STOPPED 2016-07-15 16:51:34,808 DEBUG [main] org.apache.hadoop.service.CompositeService: org.apache.hadoop.mapreduce.v2.app.MRAppMaster: stopping services, size=0

Re: Sqoop UnknownHostException when using HA namenode

owen_matthews — Wed, 20 Jul 2016 20:04:28 GMT

Just to give an update,

we found that this issue was related to a problem with kitesdk v1.0.0 properly reading the client config.

See:

https://issues.cloudera.org/browse/KITE-976

https://issues.cloudera.org/browse/KITE-1090

Our fix was to replace the following kitesdk v1.0.0 jars in sqoop/lib with the v1.1.0 versions:

kite-data-core-1.1.0.jar

kite-data-hive-1.1.0.jar

kite-data-mapreduce-1.1.0.jar

kite-hadoop-compatibility-1.1.0.jar

This resolved the UnknownHostException.

Re: Sqoop UnknownHostException when using HA namenode

dtraver — Tue, 21 Feb 2017 21:49:25 GMT

Hi @Owen,

This has been patched in the HDP 2.5.3 release.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/fixed_issues.html

BUG-65983

N/A

Increase the kite sdk version to 1.1 for fixing Sqoop parquet support issues

Regards,

@Darwin

question Re: Sqoop UnknownHostException when using HA namenode in Archives of Support Questions (Read Only)

Sqoop UnknownHostException when using HA namenode

Re: Sqoop UnknownHostException when using HA namenode

Re: Sqoop UnknownHostException when using HA namenode