CDH 5.7.1 - Same issue, but for configuring app-specific log4j:   Working spark-submit command line:  [--master yarn-cluster --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --class <class> <jar> ]   Cannot make the log4j portion above to work in Spark action - everything else is ok:   <action name="spark-7844">   <spark xmlns="uri:oozie:spark-action:0.1">     <job-tracker>${jobTracker}</job-tracker>     <name-node>${nameNode}</name-node>     <configuration>       <property>         <name>spark.driver.extraJavaOptions</name>         <value>-Dlog4j.configuration=log4j.properties</value>       </property>       <property>         <name>spark.executor.extraJavaOptions</name>         <value>-Dlog4j.configuration=log4j.properties</value>       </property>     </configuration>     <master>yarn-cluster</master>     <mode>cluster</mode>     <name>...</name>     <class>...</class>     <jar>...</jar>     <spark-opts>--executor-memory 2G --files hdfs:/user/myao/config/log4j.properties --conf &quot;spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties&quot; --conf &quot;spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties&quot; </spark-opts>   </spark>   <ok to="End"/>   <error to="Kill"/> </action>     Driver stderr:   Using properties file: null Warning: Ignoring non-spark config property: "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" Warning: Ignoring non-spark config property: "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" Parsed arguments: master yarn-cluster deployMode cluster executorMemory 2G executorCores null totalExecutorCores null propertiesFile null driverMemory null driverCores null driverExtraClassPath  ........ driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native driverExtraJavaOptions null supervise false queue null numExecutors null files hdfs:/user/myao/config/log4j.properties pyFiles null archives null mainClass .... primaryResource .... name .... childArgs [] jar  ............. packages null packagesExclusions null repositories null verbose true   Thanks!   ... View more

Thanks for your detailed reply.  That's a valid and understandable concern.  We chose Cloudera for our production Hadoop platform precisely for the quality of integration and maturity you offer.  We as users simply need some clarity from the vendor for observed feature discrepancies from the official distro, especially for such a critical component as Spark.   Are there any other discrepancy/customization that we should be aware of?  Can Cloudera be more transparent in your release notes whenever you remove/modify features from the official open-source versions?  Searching for "SparkR" in CDH5.7 release notes for Spark found 4 Jiras, which would give one the impression that SparkR is included.   Thanks again, Miles ... View more

Hortonworks HDP 2.4 includes it (v.1.6.0).   Anyway, SparkR was merged into Spark project since 1.4 (see old AmpLab project page), so I don't understand why Cloudera can't just ship it along with the rest of the Spark package.  It seems a conscious decision to remove the module - what's the reason? ... View more

No Ambari support? Cloudera Manager can manage HBase Thrift and REST servers as separate roles. Thrift server is needed by Hue HBase GUI. ... View more

javax.jdo.option.ConnectionPassword value is encrypted.  Is the clear text copy saved anywhere else, or do we have to manually reset it in the DB, then update it in CM config?   How about for other services like Oozie and Hue?     ... View more

Resetting to default didn't work in the current layout, but works in Classic - thanks!   Do I have to keep classic layout from now on, or can I switch back?   ... View more

Also, if you leave Mail Server Username/Password blank (default), it will auto-populate to CM 'admin' ID and password the next time around.  Entering a valid Unix ID/password does not help, either (the logs don't tell you whether it even get used).   'cloudera-scm' login is configured nologin, so can't test command-line mail.   ... View more

Thanks for the reply.  We tried various settings - server FQDN instead of 'localhost', different logins and From field, none worked.    Alert Publisher log:   2016-05-18 16:23:10,036 INFO com.cloudera.enterprise.alertpublisher.processor.EmailSubjectGenerator.verbose: Generated subject [Cloudera Alert] Test Alert. 2016-05-18 16:23:10,043 ERROR org.apache.camel.processor.DefaultErrorHandler: Failed delivery for exchangeId: ...... Exhausted after delivery attempt: 1 caught: org.springframework.mail.MailAuthenticationException: Authentication failed; nested exception is javax.mail.AuthenticationFailedException: No authentication mechansims supported by both server and client org.springframework.mail.MailAuthenticationException: Authentication failed; nested exception is javax.mail.AuthenticationFailedException: No authentication mechansims supported by both server and client at org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:392) at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:340) at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:355) at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:344) at org.apache.camel.component.mail.MailProducer.process(MailProducer.java:44) at org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104) at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:272) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:98) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:99) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:299) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:208) at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:269) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.Pipeline.process(Pipeline.java:125) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:109) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:189) at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:121) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: javax.mail.AuthenticationFailedException: No authentication mechansims supported by both server and client at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:590) at javax.mail.Service.connect(Service.java:313) at org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:389) ... 31 more     Error messages in /var/log/maillog look like:  "sendmail[12308]: u4ILEPlD012308: localhost [127.0.0.1] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA" Checked /etc/hosts and /etc/mail - all fairly standard setup, no explicit restrictions.   I can manually send mail from the host with [ date |mailx -s "test" <id@domain> ] from different logins.   What I see may be the cause is that the CMS processes owner is not mapped properly somehow:   111 5755 19346 1 15:46 ? 00:00:17 /usr/java/jdk1.8.0_60/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-ALERTPUBLISHER-hou706068.int.cggveritas.com.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /var/run/cloudera-scm-agent/process/504-cloudera-mgmt-ALERTPUBLISHER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.enterprise.alertpublisher.AlertPublisher   The UID maps to 'cloudera-scm' locally, but another human user globally in NIS:   /var/log/cloudera-scm-alertpublisher$ id cloudera-scm uid=111(cloudera-scm) gid=115(cloudera-scm) groups=115(cloudera-scm)   /var/log/cloudera-scm-alertpublisher$ ypcat passwd|grep ':111:' ***:uQeRidWxOqnd2:12118:111:.....     Can this be the reason?  Any workaround other than re-provision the UID?   ... View more