Reply
New Contributor
Posts: 1
Registered: ‎11-27-2017

Problem Exercise 3 spark (Hadoop tutorial)

[ Edited ]

Hello!!!

Im trying to do the exercise 3 of the clourdera tutorial : 

https://www.cloudera.com/developers/get-started-with-hadoop-tutorial/exercise-3.html

 

Im running cloudera through docker.

 

I ran spark shell with this command :  

sudo -u hdfs spark-shell --master yarn-client

 

i got a problem when running the code (given in the exercise 3) with spark shell : 

 

cooccurrences: org.apache.spark.rdd.RDD[(Int, Iterator[((String, String), Int)])] = MapPartitionsRDD[11] at map at <console>:43

scala> val combos = cooccurrences.flatMap(x => x._2).reduceByKey((a, b) => a + b)
combos: org.apache.spark.rdd.RDD[((String, String), Int)] = ShuffledRDD[13] at reduceByKey at <console>:45

scala> val mostCommon = combos.map(x => (x._2, x._1)).sortByKey(false).take(10)
[Stage 0:>                  (0 + 0) / 2][Stage 1:>                  (0 + 0) / 2]17/11/27 15:55:44 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1511783675188_0014_01_000002 on host: quickstart.cloudera. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1511783675188_0014_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
        at org.apache.hadoop.util.Shell.run(Shell.java:478)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

 

 

After this issue, the shell keeps looping on 

 

17/11/27 15:55:57 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/11/27 15:56:12 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/11/27 15:56:27 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources


Can someone helps me to solve this please??

 

Thanks a lot.

 

 

 

Announcements