Member since
04-08-2016
4
Posts
0
Kudos Received
0
Solutions
12-08-2016
07:32 PM
Hi, We are using spark in Microsoft Azure. Sometimes we get this error when we submit spark job. Its not like every time but when it comes up SparkContext doesn't get created. 16/12/08 08:02:18 INFO [akka.event.slf4j.Slf4jLogger] 80==> Slf4jLogger started
error while starting up loggers
akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.slf4j.Slf4jLogger] due to [akka.event.Logging$LoggerInitializationException: Logger log1-Slf4jLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]]
at akka.event.LoggingBus$anonfun$4$anonfun$apply$1.applyOrElse(Logging.scala:116)
at akka.event.LoggingBus$anonfun$4$anonfun$apply$1.applyOrElse(Logging.scala:115)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at akka.event.LoggingBus$anonfun$4.apply(Logging.scala:115)
at akka.event.LoggingBus$anonfun$4.apply(Logging.scala:110)
at scala.collection.TraversableLike$WithFilter$anonfun$map$2.apply(TraversableLike.scala:722)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
at akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:110)
at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:623)
at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:157)
at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$doCreateActorSystem(AkkaUtils.scala:121)
at org.apache.spark.util.AkkaUtils$anonfun$1.apply(AkkaUtils.scala:53)
at org.apache.spark.util.AkkaUtils$anonfun$1.apply(AkkaUtils.scala:52)
at org.apache.spark.util.Utils$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1988)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1979)
at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:266)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:288)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:457)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2304) Thanks Adnan Ahmed
... View more
Labels:
04-25-2016
10:38 PM
Hi Benjamin, I tested that in my local and Hortonworks sandbox. Both places I get expected behavior, it is based on splits size. I think it is something related to wasb. Thanks
... View more
04-25-2016
05:01 PM
I am getting a strange behavior. I have a file stored in azure wasb (size 1 GB) when I create an RDD using below statement, it only creates two partitions. I am under impression it should be based on HDFS block size which is 128M in our environment. val fileRDD = sc.textFile("/user/aahmed/file.csv") Seems like its creates one partition for 500MB each. I tried it with one large file (28G) and I got 56 partitions. It supposed to be based HDFS block size not based on 500MB
... View more
Labels:
04-08-2016
03:24 PM
When I submit a spark job using below command, spark-submit --num-executors 10 --executor-cores 5 --executor-memory 2G --master yarn-cluster --conf spark.driver.userClassPathFirst=true --conf spark.executor.userClassPathFirst=true --class com.example.SparkJob target/scala-2.10/spark-poc-assembly-0.1.jar 10.0.201.6 hdfs:///user/aahmed/example.csv It gives me these messages on console. I want to see org.apache.spark INFO level message. How and where can I configure this? 16/04/08 15:09:50 INFO Client: Application report for application_1460098549233_0013 (state: RUNNING) 16/04/08 15:09:51 INFO Client: Application report for application_1460098549233_0013 (state: RUNNING) 16/04/08 15:09:52 INFO Client: Application report for application_1460098549233_0013 (state: RUNNING) 16/04/08 15:09:53 INFO Client: Application report for application_1460098549233_0013 (state: RUNNING) 16/04/08 15:09:54 INFO Client: Application report for application_1460098549233_0013 (state: RUNNING)
... View more
Labels: