@ishika as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

Hi Yuexin   Did you manage to get any resolution for this? I am able to run spark job as a --proxy-user under yarn cluster mode. However I can successfully run using yarn-client mode. This is when using CDH 6.2.1 version of Spark.   There is no problem when using opensource version of Spark with --proxy-user either on client or cluster mode.   Thanks ... View more

Hi,   This parameter spark.executor.memory (or) spark.yarn.executor.memoryOverhead can be set in Spark submit command or you can set it Advanced configurations.   Thanks AKR ... View more

Hi,   Check for the total no of applications in the Application history path, if the total no of files is more try to increase the heap size and look whether it works. Alternatively look for the spark history server logs too for any errors.   Thanks AKR ... View more

Hi Sara, I run vulnerability scans and our scanner picking up Spark 1.6 banner from the following path- for CVE-2018-8024 vulnerability, you did mention this vulnerability doesn't affect SPark 1.6 but didn't give detail reasons. This is where Qualys picksup the banner- /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/spark/conf/spark-env.sh: line 75: /usr/appl/cloudera/java/jdk1.8.0_162: is a directory Welcome to ____              __ / __/__  ___ _____/ /__ _\ \/ _ \/ _ `/ __/  '_/ /___/ .__/\_,_/_/ /_/\_\   version 1.6.0 /_/   Also we have 2 versions of SPARK running- do you really need version 1.6 to run v 2.3.0 Can you please help, advise.   ... View more

Depends on what you mean by 'storage locations'. If you mean "can other apps use HDFS?" then the answer is yes, as HDFS is an independent system unrelated to YARN and has its own access and control mechanisms not governed by a YARN scheduler. If you mean "can other apps use the scratch space on NM nodes" then the answer is no, as only local containers get to use that. If you're looking to strictly split both storage and compute, as opposed to just some form of compute, then it may be better to divide up the cluster entirely. ... View more

Hi    Thanks for posting the question to cloudera commuinity.   It would be a little too vargue to say how to check the performance. There are two things I'd suggest to do:   1) Benchmark the performance with the built-in load testing tools such as ConsumerPerformance and EndToEndLatecy.   2) Monitor the Kafka performance with the avaiable metrics: https://www.cloudera.com/documentation/kafka/latest/topics/kafka_metrics.html   Additionally, you may want to check the cloudera documentations on how to tune kafka broker, consumer and producers for better performance:   https://www.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html   Hope above information helps. ... View more

The error message shows you don't have a valid leader for the partition you are accessing. In kafka, all read/writes should go through the leader of that partition.   You should make sure the topic/partitions have healthy leader first, run: kafka-topics --describe  --zookeeper <zk_url, put /chroot if you have any> --topic <topic_name>     ... View more

Dynamic Allocation [1] controls the number of parallel running executor containers in YARN, not the number of CPU vcore resource allocated to a single executor container. [1] - https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation ... View more

Hello, I am facing the same problem, could you help me, and give me more detail, I will appreciate it.   Thanks in advance. ... View more

The error message says spark can't find the opennlp class:   opennlp.tools.stemmer.PorterStemmer   You need to check if the opennlp-tools-1.6.0.jar is present. If not, you need to correct the script: # get openNLP jar wget http://apache.mirrors.hoobly.com/opennlp/opennlp-1.6.0/apache-opennlp-1.6.0-bin.zip unzip apache-opennlp-1.6.0-bin.zip cp apache-opennlp-1.6.0/lib/opennlp-tools-1.6.0.jar .   ... View more

You may need to increase:  yarn.nodemanager.resource.memory-mb yarn.scheduler.maximum-allocation-mb   the default one could be too small to launch a default spark executor container ( 1024MB + 512 overhead).   You may also want to enable INFO logging for the spark shell to understand what exact error/warn it has: /etc/spark/conf/log4j.properties ... View more

I posted an issue yesterday that relates to this -- the spark-submit classpath seems to conflict with commons-compress from a suppiled uber-jar.   I've tried the --conf, --jar, and the --packages flags with spark-submit with no resolution.     Spark 2.x + Tika: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamF   Any help would be greatly appreciated!!!!     ... View more

CDH spark packed in CDH 6.0 is based on Apache Spark 2.2, cloudera then adds fixes/features which are improtant for its customers:   https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_60_packaging.html#cdh_packaging_600 ... View more

You can have multiple flume agents running on multiple hosts. If they are sharing the same flume configured group.id, the messages will be distributed to all the agents (not duplicated). If you don't need to do any processing on the events (via an interceptor), you could just use a kafka channel and hdfs sink, and that would deliver events directly from the channel. In that case you can only use one topic per channel, but could then have an associated sink delivering just that topic. If you did want to use a flume kafka source, it adds a 'topic' header that specifies the topic name that the message was consumed from, and you could put that in the hdfs.path or hdfs.filePrefix as %{topic} -pd ... View more

Spark Streaming in CDH 5.14 uses Apache 0.10.2 based kafka clients, that is Cloudera Kafka 2.2.0, you can find the spark gateway setup the classpath through /etc/spark/conf/classpath.txt :     [root@host-514 ~]# hadoop version Hadoop 2.6.0-cdh5.14.2 Subversion http://github.com/cloudera/hadoop -r 5724a4ad7a27f7af31aa725694d3df09a68bb213 Compiled by jenkins on 2018-03-27T20:40Z Compiled with protoc 2.5.0 From source with checksum 302899e86485742c090f626a828b28 This command was run using /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hadoop-common-2.6.0-cdh5.14.2.jar [root@host-514 ~]# cat /etc/spark/conf/classpath.txt |grep kafka |grep -v flume /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/kafka-clients-0.10.2-kafka-2.2.0.jar /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/kafka_2.10-0.10.2-kafka-2.2.0.jar     ... View more

We do not have plan to provide RPM installer for spark2 on CDH 5.  We may provide both RPM and parcels when we release CDH 6. ... View more

Hi  Yuexin,   Thanks for your response. I have gone through all these links and many more to research on this issue. And I have already done all this configuration of setting jaas file for both driver and executor and also setting kafka ssl settings in the kafkaparams in the program.   $SPARK_HOME/bin/spark-submit \             --conf spark.yarn.queue=$yarnQueue \             --conf spark.hadoop.yarn.timeline-service.enabled=false \             --conf spark.yarn.archive=$sparkYarnArchive \             $sparkOpts \             --properties-file $sparkPropertiesFile \             --files /conf/kafka/kafka_client_jaas_dev.conf, /conf/ kafka/krb5_dev.conf, /conf/ keystore/ kafka_client_truststore.jks ,conf/kafka/kafka/kafkausr.keytab \             --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config= kafka_client_jaas_dev.conf -Djava.security.krb5.conf=krb5_dev.conf -Dsun.security.krb5.debug=true" \             --driver-java-options "-Djava.security.auth.login.config=/conf/ kafka_client_jaas_dev.conf  -Djava.security.krb5.conf=/conf/ krb5_dev.conf  \             --class com.commerzbank.streams.KafkaHDFSPersister $moduleJar \             $1 $2 $3 $4 $KafkaParamsconfFile \   Problem here is running in yarn client mode (and all links you mentioned also talks about yarn cluster mode) but i have to run in yarn client mode only here due to some project constraints , and issue is if I specify full path of keystore file(where it is located on edge node where I run the command)  in 'ssl.truststore.location' parameter then executors cannot find this file in their cache as it looks for complete path+file name and executor cache contains file with name 'kafka_client_truststore.jks'.   And when i pass the keystore file (kafka_client_truststore.jks) without path to  'ssl.truststore.location' parameter then it fails for driver as driver looks in current path of edge node from where the job is run (and if I run the job from same directory /conf/keystore where this keystore file is present on edge node, then job succeeds)   Is there way to solve this in your view or better way to load same set of files for driver and executors running in yarn client mode.    Regards, Hitesh ... View more

This is usually caused by not having proper HADOOP or SPARK CONF on the node. You need to assign spark2 gateway role to this node, and deploy spark2 client configureations, then re-launch spark2-shell. ... View more

Hi Suku: I response some of your questions: a) Which Keytab you have used, whether CM generated keytab or user keytab generated by you? I used kafka.keytab b) Path of your jaas.conf and keytab for Kafka? Path of kafka.keytab in /etc/security/keytabs/ c) How Kafka Kerberos configuration parameters set? The following is the configuration of Kafka parameters and the the form to use the jaas parameter. Properties props = new Properties(); props.put("bootstrap.servers", "xxxx:9092,xxx:9092"); props.put("client.id", "client-id-coprocessor "); props.put("key.serializer", StringSerializer.class.getName()); props.put("value.serializer", StringSerializer.class.getName()); props.put("security.protocol", "SASL_PLAINTEXT"); props.put("sasl.kerberos.service.name", "kafka"); props.put("sasl.jaas.config", "com.sun.security.auth.module.Krb5LoginModule required \n" + "useKeyTab=true \n" + "storeKey=true \n" + "keyTab=\"/etc/security/keytabs/kafka.keytab\" \n" + "principal=\"kafka/nodo@REALM\";"); KafkaProducer producer = new KafkaProducerString>(props); Remember sometimes you will need reboot your hbase service for deploy your coprocessor. I hope I will help you. Florentino ... View more

We have validated the safety valve, as well as the solr URL.  It seems that someone had deployed some examples a long timago and never cleaned them up so we had old information stored in collections listed in zookeeper.  We deleted those and ran solrctrl init --force and that seemed to resolve the issue.  We still however see the old dashboard entries in the search menu in Hue and they remain even after deleting them from the Hue database.  We have an open case now for it. ... View more

This is something I found on Stackoverflow..   https://stackoverflow.com/questions/36508553/how-to-specify-consumer-group-in-kafka-spark-streaming-using-direct-stream.   Is there any other way of monitoring the backlog in this approach. I tried exploring Spark Metrics API. Even it is of no big use in this case. ... View more

This could be caused by two possible situations:   1) all of you kafka brokers are down (unlikely) 2) you are not connecting to the correct zookeeper, or specified a wrong root zk node(most likely). Please check if you really need /kafka in your zookeper conenct string, check your "zookeeper.chroot" configuraion on CM. If it's empty there, you don't need to append /kafka in zookeeper connect string, or if's something else, replace with it. ... View more

Glad the issue the resolved but try to use some other location to store those jars instead of /opt/cloudera/parcels as when you upgrade you can lose all those jars. ... View more

i was pointinn to a wrong broker so the zk was unable to find the topics ... View more