11-27-2017 08:17 AM - last edited on 11-28-2017 06:02 AM by cjervis
Im trying to do the exercise 3 of the clourdera tutorial :
Im running cloudera through docker.
I ran spark shell with this command :
sudo -u hdfs spark-shell --master yarn-client
i got a problem when running the code (given in the exercise 3) with spark shell :
cooccurrences: org.apache.spark.rdd.RDD[(Int, Iterator[((String, String), Int)])] = MapPartitionsRDD at map at <console>:43 scala> val combos = cooccurrences.flatMap(x => x._2).reduceByKey((a, b) => a + b) combos: org.apache.spark.rdd.RDD[((String, String), Int)] = ShuffledRDD at reduceByKey at <console>:45 scala> val mostCommon = combos.map(x => (x._2, x._1)).sortByKey(false).take(10) [Stage 0:> (0 + 0) / 2][Stage 1:> (0 + 0) / 2]17/11/27 15:55:44 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1511783675188_0014_01_000002 on host: quickstart.cloudera. Exit status: 1. Diagnostics: Exception from container-launch. Container id: container_1511783675188_0014_01_000002 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:561) at org.apache.hadoop.util.Shell.run(Shell.java:478) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1
After this issue, the shell keeps looping on
17/11/27 15:55:57 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 17/11/27 15:56:12 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 17/11/27 15:56:27 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Can someone helps me to solve this please??
Thanks a lot.