We're getting an UnknownHostException when running Sqoop.
The job gets submitted successfully to the cluster, but then fails almost immediately.
In the job history logs, we can see the exception stack trace below.
Background details:
- client OS is RHEL 6.7
- Sqoop version is 1.4.6.2.3.2.0-2950
- Our cluster is running HDP 2.4.2
- Our cluster is configured with an HA namenode, called 'titan'.
- We're using Kerberos for authentication.
- We can run other MapReduce and Spark jobs successfully, it is just Sqoop that is failing with this error.
- We can run this Sqoop job on an older (HDP 2.3.2) cluster successfully (it also uses HA namenode).
Stacktrace:
2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
2016-07-15 16:51:34,762 DEBUG [main] org.apache.hadoop.hdfs.BlockReaderLocal: dfs.domain.socket.path =
2016-07-15 16:51:34,807 DEBUG [main] org.apache.hadoop.service.AbstractService: noteFailure org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan
2016-07-15 16:51:34,807 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:519)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:499)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1598)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:499)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:285)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1556)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: titan
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:66)
at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:50)
at org.kitesdk.data.spi.Registration.lookupDatasetUri(Registration.java:106)
at org.kitesdk.data.Datasets.load(Datasets.java:103)
at org.kitesdk.data.Datasets.load(Datasets.java:165)
at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510)
at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:517)
... 11 more
Caused by: java.net.UnknownHostException: titan
... 30 more
2016-07-15 16:51:34,808 DEBUG [main] org.apache.hadoop.service.AbstractService: Service: org.apache.hadoop.mapreduce.v2.app.MRAppMaster entered state STOPPED
2016-07-15 16:51:34,808 DEBUG [main] org.apache.hadoop.service.CompositeService: org.apache.hadoop.mapreduce.v2.app.MRAppMaster: stopping services, size=0
