Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot run MapReduce with azure blob storage (WASB) - UPDATE

Cannot run MapReduce with azure blob storage (WASB) - UPDATE

New Contributor

We are trying to run smoke test mapreduce job from hadoop examples (terasort):

[hdfs@xxxxx 2.2.0.0-2041]$ hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar teragen 100 /test/10gsort/input

We are dealing with following problems:

[hdfs@xxxxx bduser]$ hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar teragen 100 /test/10gsort/input
16/01/28 09:44:55 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
16/01/28 09:44:56 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
16/01/28 09:44:56 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
16/01/28 09:44:57 INFO client.RMProxy: Connecting to ResourceManager at xxxxx/10.0.0.8:8050
16/01/28 09:44:58 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: java.lang.reflect.InvocationTargetException
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:305)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:309)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
16/01/28 09:44:58 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
16/01/28 09:44:58 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
16/01/28 09:44:58 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.



6 REPLIES 6

Re: Cannot run MapReduce with azure blob storage (WASB) - UPDATE

Re: Cannot run MapReduce with azure blob storage (WASB) - UPDATE

New Contributor

I've checked the setup and moved a little bit, please see update.


					
				
			
			
				
			
			
			
			
			
			
			
		

Re: Cannot run MapReduce with azure blob storage (WASB) - UPDATE

@Martin Repka Please update the thread

Re: Cannot run MapReduce with azure blob storage (WASB) - UPDATE

Mentor

@Martin Repka can you confirm all of the properties in the following document on Azure are set? link

Re: Cannot run MapReduce with azure blob storage (WASB) - UPDATE

New Contributor

Properties look set correctly due to the document.

Re: Cannot run MapReduce with azure blob storage (WASB) - UPDATE

Mentor
@Martin Repka

just out of curiousity, what HDP version are you on? I see you're executing

/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar

do you have HDP 2.3.4 for example? I'm just wondering if there are some bugs that were addressed in later versions.

Don't have an account?
Coming from Hortonworks? Activate your account here