Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Config Yarn with cluster NameNode

Solved Go to solution

Config Yarn with cluster NameNode

Explorer

My system have

2 NameNode as Cluster (nn1 active, nn2 standby)

2 DataNode

 

1 Yarn Resource Manager service is on nn1

2 Yarn Node Manager is on 2 DataNode

 

Before config cluster NameNode everything work well

After config Namde Node as cluster , Yarn start ok but when submit script from Hue I have some error

 

2018-03-15 15:46:49,203 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download resource { { hdfs://mycluster/user/oozie/share/lib/lib_20171215093741/pig/jsp-2.1-6.1.14.jar, 1513305464001, FILE, null },pending,[(container_1521103549281_0001_02_000001)],8742037577451777,DOWNLOADING}
java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:728)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:671)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:155)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:249)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: mycluster

View log on both Yarn Node manager service I see above error .

How Can I fix this

Thanks

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Config Yarn with cluster NameNode

Contributor

Have a look at your HDFS configuration and the name you are using for Name Node service. Confirm that you are using correct name for Name Node service. You can look at your hdfs-site.xml file on your  deployed Hue server to confirm.

2 REPLIES 2

Re: Config Yarn with cluster NameNode

Contributor

Have a look at your HDFS configuration and the name you are using for Name Node service. Confirm that you are using correct name for Name Node service. You can look at your hdfs-site.xml file on your  deployed Hue server to confirm.

Highlighted

Re: Config Yarn with cluster NameNode

Explorer

After check again hdfs-site.xml on 2 DataNode where Yarn Node Manager stand on , I see that the hdfs-site file missing this line when compare with the hdfs-site on Name Node

 

<property>
  <name>dfs.client.failover.proxy.provider.radcluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

restart the Yarn Node Manager Service , It work now , Thanks you

Don't have an account?
Coming from Hortonworks? Activate your account here