Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I am building a cluster to have Jenkins>Kafka>Spark>Cassandra to work together

I am building a cluster to have Jenkins>Kafka>Spark>Cassandra to work together

Contributor

Demo Cluster

I am building a cluster to have Jenkins>Kafka>Spark>Cassandra to work together. We can call it as Integration cluster.

 

I am facing issue while running build job from Jenkins, Spark fails to receive messages from Kafka. It generates error message attached below. Please suggest what could be wrong.

 

Env: Cloudera Manager, CDH5.4.4 , Spark 1.3.0 is on YARN Client mode, Debain x64-bit OS, Jenkins, Kafka broker only, Cassandra write database, Impala as read database.

Namenode: 32GB RAM, Xenon 8 cores CPU, 10 GB/s network

Datanode: 24 GB RAM, Xenon 4 cores CPU

 

Please suggest to slove the issue, I am stuck.  What could be wrong. CM Dashboard looks fine doesn't show up any configuration issues. Screenshot below

Image.png

 

ERROR

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 75, datanode03.rnd.company.net): ExecutorLostFailure (executor 2 lost)

Driver stacktrace:

        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)

        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)

        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)

        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)

        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)

        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)

        at scala.Option.foreach(Option.scala:236)

        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)

        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

 

 

1 REPLY 1

Re: I am building a cluster to have Jenkins>Kafka>Spark>Cassandra to work together

Master Guru
The driver log does not tell us much other than that it tried to run a task under an executor about 4 times but every attempt failed (it does not include the reason why).

Could you check your executor logs instead? Its usually visible in the Spark History Server Web UI, after you click through to your application and then visit the Executors tab and click stderr. Could you post that log from the failing application? It would have the reason of why the tasks that went onto it may have failed.