Does anyone have any idea why in an application where no code has changed, I get the error message below? Is the machine running out of resources? Is there something that could have gone corrupted? I am only starting a process that 'used to work with no errors', before indeed, the Hadoop developer left, and the management did not hire a new person with this knowledge. 22/05/16 05:08:01 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164) at org.apache.spark.SparkContext.<init>(SparkContext.scala:500) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:933) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:924) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:924) at com.olbico.spark.JobManager$class.sparkSession(JobManager.scala:40) at com.olbico.spark.MergeManager$.sparkSession$lzycompute(MergeManager.scala:27) at com.olbico.spark.MergeManager$.sparkSession(MergeManager.scala:27) at com.olbico.spark.MergeManager$.Execute(MergeManager.scala:111) at com.olbico.spark.MergeManager$.main(MergeManager.scala:92) at com.olbico.spark.MergeManager.main(MergeManager.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
... View more
I am sorry I am very novice and I am the only one engineer that had to take over a nutch crawling process. My apologies in advance for my vague question. My problem is the following. We start a crawling process on a hadoop cluster from a pentaho/spoon job on a windows machine. That spoon.job exits with a failure, but the yarn applications keep on starting, and I have no idea where they come from or how to stop them. So basically a few yarn mapreduce type applications are started in a loop: partition, fetch, crawldb, UpdateHostDb. I can kill the current application, but even then the yarn application I killed restarts and they keep appearing from something that has started them. That 'something' either being on the hadoop cluster or on windows, I have no idea. I am lost. Hence my question: What starts the yarn applications? And how do I kill the process that starts these yarn applications? I now have to wait like 3 days until that loop is finished to try again. And I don't really know why things are failing so, that is 2 tries, one week further. People mad at me.
... View more