Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark metric.properties file on CDH 6.3 is deleted

Highlighted

Spark metric.properties file on CDH 6.3 is deleted

Expert Contributor

I am trying to follow https://github.com/ibm-research-ireland/sparkoscope Enabling Spark Optimization through Cross-stack Monitoring and Visualization. I can't find spark.properties configuration on Cloudera Manager or as a file in /etc/spark/conf. So I created a file named metric.properties and put metrics in it as following

 

executor.source.jvm.class=org.apache.spark.metrics.source.SigarSource

executor.sink.hdfs.class=org.apache.spark.metrics.sink.HDFSSink
executor.sink.hdfs.pollPeriod = 1
executor.sink.hdfs.dir = hdfs://cloudera.impketra.com:8020/custom-metrics
executor.sink.hdfs.unit = seconds

executor.sink.mqtt.class=org.apache.spark.metrics.sink.MQTTSink
executor.sink.mqtt.pollPeriod = 1
executor.sink.mqtt.host = cloudera.impketra.com
executor.sink.mqtt.port = 1883
executor.sink.mqtt.unit = seconds

 

But after a while ( I think after restarting Spark service) the metrics.properties file is deleted.

Additionally when I submit 

 

spark-submit --class org.apache.spark.examples.SparkPi  --master yarn /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/spark-examples_2.11-2.4.0-cdh6.3.1.jar 10000

 

I get this ERROR

 

20/03/05 10:17:05 ERROR metrics.MetricsSystem: Sink class org.apache.spark.metrics.sink.MQTTSink cannot be instantiated
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.metrics.sink.MQTTSink
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:242)
        at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198)
        at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
        at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:194)
        at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:102)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:365)
        at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:201)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:221)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        ... 4 more

 

 

2 REPLIES 2

Re: Spark metric.properties file on CDH 6.3 is deleted

Expert Contributor

It looks like the Class  "org.apache.spark.metrics.sink.MQTTSink" is missing on your classpath.

 

Could you please check if you can add the corresponding Jar which holds the same class in your Driver, Executor classpath ?

Thanks,
Satz
Highlighted

Re: Spark metric.properties file on CDH 6.3 is deleted

Expert Contributor

Thank you when I copy  sigar-1.6.4.jar to /opt/cloudera/parcels/CDH/lib/spark/jars/ error disappeared. But deleting the metrics.properties file still a problem.

Don't have an account?
Coming from Hortonworks? Activate your account here