Member since
03-04-2015
96
Posts
12
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7070 | 01-04-2017 02:33 PM | |
14317 | 07-17-2015 03:11 PM |
11-03-2016
03:01 PM
The CSD download link given actually points to a jar file with the following directory tree:
$ jar tvf SPARK2_ON_YARN-2.0.0.cloudera.beta1.jar 0 Wed Sep 21 17:24:28 CDT 2016 descriptor/ 0 Wed Sep 21 17:24:28 CDT 2016 scripts/ 0 Wed Sep 21 17:24:28 CDT 2016 aux/ 0 Wed Sep 21 17:24:28 CDT 2016 aux/client/ 3312 Wed Sep 21 17:24:28 CDT 2016 images/icon.png 1711 Wed Sep 21 17:24:28 CDT 2016 aux/client/spark-env.sh 0 Wed Sep 21 17:24:28 CDT 2016 images/ 18456 Wed Sep 21 17:24:28 CDT 2016 descriptor/service.sdl 0 Wed Sep 21 17:50:46 CDT 2016 meta/ 20 Wed Sep 21 17:50:46 CDT 2016 meta/version 0 Wed Sep 21 17:50:58 CDT 2016 META-INF/ 1813 Wed Sep 21 17:24:28 CDT 2016 scripts/control.sh 12362 Wed Sep 21 17:24:28 CDT 2016 scripts/common.sh 104 Wed Sep 21 17:50:58 CDT 2016 META-INF/MANIFEST.MF
Now, when the documentation specifies "install Spark2 CSD", which file(s) is it referring to exactly? Just descriptor/service.sdl, or the entire jar to /opt/cloudera/csd? The two scripts above look like operational scripts for CM.
Thanks,
Miles Yao
... View more
Labels:
- Labels:
-
Apache Spark
09-15-2016
12:26 PM
CDH 5.7.1 - Same issue, but for configuring app-specific log4j: Working spark-submit command line: [--master yarn-cluster --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --class <class> <jar> ] Cannot make the log4j portion above to work in Spark action - everything else is ok: <action name="spark-7844"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>spark.driver.extraJavaOptions</name> <value>-Dlog4j.configuration=log4j.properties</value> </property> <property> <name>spark.executor.extraJavaOptions</name> <value>-Dlog4j.configuration=log4j.properties</value> </property> </configuration> <master>yarn-cluster</master> <mode>cluster</mode> <name>...</name> <class>...</class> <jar>...</jar> <spark-opts>--executor-memory 2G --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" </spark-opts> </spark> <ok to="End"/> <error to="Kill"/> </action> Driver stderr: Using properties file: null Warning: Ignoring non-spark config property: "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" Warning: Ignoring non-spark config property: "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" Parsed arguments: master yarn-cluster deployMode cluster executorMemory 2G executorCores null totalExecutorCores null propertiesFile null driverMemory null driverCores null driverExtraClassPath ........ driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native driverExtraJavaOptions null supervise false queue null numExecutors null files hdfs:/user/myao/config/log4j.properties pyFiles null archives null mainClass .... primaryResource .... name .... childArgs [] jar ............. packages null packagesExclusions null repositories null verbose true Thanks!
... View more
08-26-2016
09:09 AM
Thanks for your detailed reply. That's a valid and understandable concern. We chose Cloudera for our production Hadoop platform precisely for the quality of integration and maturity you offer. We as users simply need some clarity from the vendor for observed feature discrepancies from the official distro, especially for such a critical component as Spark. Are there any other discrepancy/customization that we should be aware of? Can Cloudera be more transparent in your release notes whenever you remove/modify features from the official open-source versions? Searching for "SparkR" in CDH5.7 release notes for Spark found 4 Jiras, which would give one the impression that SparkR is included. Thanks again, Miles
... View more
08-25-2016
09:50 AM
Hortonworks HDP 2.4 includes it (v.1.6.0). Anyway, SparkR was merged into Spark project since 1.4 (see old AmpLab project page), so I don't understand why Cloudera can't just ship it along with the rest of the Spark package. It seems a conscious decision to remove the module - what's the reason?
... View more
06-24-2016
09:19 PM
No Ambari support? Cloudera Manager can manage HBase Thrift and REST servers as separate roles.
Thrift server is needed by Hue HBase GUI.
... View more
06-23-2016
08:55 AM
javax.jdo.option.ConnectionPassword value is encrypted. Is the clear text copy saved anywhere else, or do we have to manually reset it in the DB, then update it in CM config? How about for other services like Oozie and Hue?
... View more
06-02-2016
02:42 PM
CDH 5.8 seems to be on the horizon, according to the recent Hue blog. Can the good company folks provide some heads-up about the rough release date, and the main feature set? This will be very helpful for our planning purpose, as we are considering upgrade to 5.7. Thanks! Miles Yao
... View more
Labels:
05-18-2016
03:35 PM
Resetting to default didn't work in the current layout, but works in Classic - thanks! Do I have to keep classic layout from now on, or can I switch back?
... View more
05-18-2016
02:48 PM
Also, if you leave Mail Server Username/Password blank (default), it will auto-populate to CM 'admin' ID and password the next time around. Entering a valid Unix ID/password does not help, either (the logs don't tell you whether it even get used). 'cloudera-scm' login is configured nologin, so can't test command-line mail.
... View more
05-18-2016
02:35 PM
Thanks for the reply. We tried various settings - server FQDN instead of 'localhost', different logins and From field, none worked. Alert Publisher log: 2016-05-18 16:23:10,036 INFO com.cloudera.enterprise.alertpublisher.processor.EmailSubjectGenerator.verbose: Generated subject [Cloudera Alert] Test Alert. 2016-05-18 16:23:10,043 ERROR org.apache.camel.processor.DefaultErrorHandler: Failed delivery for exchangeId: ...... Exhausted after delivery attempt: 1 caught: org.springframework.mail.MailAuthenticationException: Authentication failed; nested exception is javax.mail.AuthenticationFailedException: No authentication mechansims supported by both server and client org.springframework.mail.MailAuthenticationException: Authentication failed; nested exception is javax.mail.AuthenticationFailedException: No authentication mechansims supported by both server and client at org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:392) at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:340) at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:355) at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:344) at org.apache.camel.component.mail.MailProducer.process(MailProducer.java:44) at org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104) at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:272) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:98) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:99) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:299) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:208) at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:269) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.Pipeline.process(Pipeline.java:125) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:109) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:189) at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:121) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: javax.mail.AuthenticationFailedException: No authentication mechansims supported by both server and client at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:590) at javax.mail.Service.connect(Service.java:313) at org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:389) ... 31 more Error messages in /var/log/maillog look like: "sendmail[12308]: u4ILEPlD012308: localhost [127.0.0.1] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA" Checked /etc/hosts and /etc/mail - all fairly standard setup, no explicit restrictions. I can manually send mail from the host with [ date |mailx -s "test" <id@domain> ] from different logins. What I see may be the cause is that the CMS processes owner is not mapped properly somehow: 111 5755 19346 1 15:46 ? 00:00:17 /usr/java/jdk1.8.0_60/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-ALERTPUBLISHER-hou706068.int.cggveritas.com.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /var/run/cloudera-scm-agent/process/504-cloudera-mgmt-ALERTPUBLISHER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.enterprise.alertpublisher.AlertPublisher The UID maps to 'cloudera-scm' locally, but another human user globally in NIS: /var/log/cloudera-scm-alertpublisher$ id cloudera-scm uid=111(cloudera-scm) gid=115(cloudera-scm) groups=115(cloudera-scm) /var/log/cloudera-scm-alertpublisher$ ypcat passwd|grep ':111:' ***:uQeRidWxOqnd2:12118:111:..... Can this be the reason? Any workaround other than re-provision the UID?
... View more