It looks like the Class  "org.apache.spark.metrics.sink.MQTTSink" is missing on your classpath.   Could you please check if you can add the corresponding Jar which holds the same class in your Driver, Executor classpath ? ... View more

Hello @lakshmipathy ,   Please refer the below thread, hope this would help   https://community.cloudera.com/t5/Support-Questions/How-to-define-topic-retention-with-kafka/td-p/222671 ... View more

Hello @ravisro ,   I don't think there would be a straight forward way for this, (i.e) we might need to perform some sort of data cleansing work prior feeding it to spark in my view. Possibly the inputs shared does contains new line characters (\n) which might make spark to confuse with the data and new lines.    I did some sort of data cleansing, (i.e) removing newlines gave me below result   inputDf . show ( false ) +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+ |_c0 |_c1 |_c2 | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+ |2020-02-23 11:15:39|"Hi Craig, Please approve the standard pricing. No further amendments made "Legal System."Justification -XXX is the sole owner in China Thank you."|Approved| +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+  while running the same code from spark. ... View more

Hello @Dylan-    Is your CSV files are in Git and you are pulling into your project through CDSW ? How big your CSV files are? ... View more

Hello @RichardJG ,   Thanks for posting your query.   Based on the documentation, at present Kafka connect is not supported   https://www.cloudera.com/documentation/kafka/latest/topics/kafka_known_issues.html#xd_583c10bfdbd326ba-590cb1d1-149e9ca9886--6fcb__section_ens_4bf_55   Is it possible for you to post the complete exception? ... View more

Hello @Sankar   Thanks for posting your query,   You can try launching R engine in your CDSW and try running your code.   Also, could you please share us what was the error which you are receiving while launching the job and how you are running the job ?   ... View more

Hello @desind   From the Message, it looks the SSL enabled URL is giving connection timeout.   Could you try running the curl with -vvv option and see what are the request responses ? ... View more

Hello @regeamor   Thank you for posting the query with us.   Basically when you enable Dynamic allocation it gracefully remove the Idle containers which were idle for (60s as default) But when you lower the value will remove the executors frequently depending upon the executor's usage (just like the tasks getting allocated to those executors and the amount of tasks required)   https://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors   Also, the above said behaviour is upto Spark (and yarn container pre-emption wont be aware of it) ... View more

Hello @Alfred_Lo,   Thank you for posting your query.   I could see you are using Oozie job and the job which you are referring as "SUCCEEDED" is oozie launcher job.    So, the action which was triggered from oozie launcher got failed (Example: Your MR action, yarn action, spark action etc., ) but the oozie launcher succeeded and hence you are seeing the status of oozie launcher job as "SUCCEEDED"     ... View more

Hello @Sankar   Thanks for posting your query with us!   From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager   Does your normal spark jobs from CDSW is running fine ? ... View more

Hello @yasmin,   Thanks for reaching us !   As per your reply it says that user who has triggered job, his user id is not listed on that particular node ? Is my understanding correct? ==> Yes you're right !   And your query regarding job slowness, we should consider the factors you mentioned along with below messages as well   ~~~ ExecutorLostFailure (executor 1480 exited caused by one of the running tasks) Reason: Stale executor after cluster manager re-registered.   I had 438 failures on 869 taks, that is a huge rate, another part has 873 out of 1236 ~~~   Here it seems the executors are getting lost and as a results tasks were dying.   Could you please check the yarn logs for the application (by using #yarn logs command in my previous reply) and see if there is any errors in executor logs. This will help us to see if there is any spcific reason for executor failures   Are you running  job in spark-client mode or cluster mode?   ... View more

Yeah, thats right. There is some issue while rendering the images on your browser. Could you please try checking in Javascript console on any of your browser while running the snippet   you may refer below URL for opening Javascript consoles:   https://webmasters.stackexchange.com/questions/8525/how-do-i-open-the-javascript-console-in-different-browsers ... View more

Hello @dpugazhe   Thanks for posting your query.   From the message   "/opt/cloudera/parcels/CDSW-1.2.2.p1.216803/scripts/cdsw-pod-logs.sh: line 40: kubectl: command not found"   It seems your installation is missing the necessary kubernetes package. Could you please check if there was any installation errors on the node which you specified ?     ... View more

Hello @yasmin   Thanks for posting your query.   From the error message attached I see the Nodemanager node is complaining that the user ID is not present on the node. Please check if the username is present on the node (where AM is running) or if you are using AD/LDAP just make sure the node is able to resolve the particular username    You can also check more on container logs, by running    #yarn logs -applicationId <applicationID> -appOwner <username_who_triggered_job>    ... View more

Hello Priya,   Thanks for posting your query!   Could you please check the spark thrift service's heap configuration ? ( ps -axu | grep <pid_of_sparkthrift ) you should be able to see Xmx options passed as argument   Consider increasing the heap of the same ... View more

Hello @upasana   Could you please share the complete log / exception trace, so that we can see from where you are getting that error   ... View more

Hello @corestack,   Thanks for posting your query    I am assuming you are talking about Spark-on-YARN (please correct me if I'm wrong), if so you may use Resource Manager APIs.    https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_Statistics_API   You can user Spark history server APIs too      https://spark.apache.org/docs/2.2.0/monitoring.html#rest-api ... View more

Hello @heena   You can try them by using escape characters "\" in front of them:   Example   "[" can be written as "\[" ... View more

@mchia Thanks for your update   From your input I believe you added a space between the username and Group name (which is correct)    could you please try below setting once and let us know   1) Cloudera Manager->Yarn->Configuration > yarn.admin.acls  yarn,hue <your_admin_group> Note there is a space between the list of users and the list of groups (in above list)   2) Cloudera Manager==>Yarn==>Configuration==>MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml  <property>  <name>mapreduce.cluster.acls.enabled</name>  <value>true</value>  </property>  <property>  <name>mapreduce.cluster.administrators</name>  <value>yarn,hue <your_admin_group></value>  </property>  <property>  <name>mapreduce.job.acl-view-job</name>  <value>yarn,hue <your_admin_group></value>  </property>    Note there is a space between the list of users and the list of groups (in above list)   3) Cloudera Manager==>Yarn==>Configuration==>YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)  <property>  <name>mapreduce.cluster.acls.enabled</name>  <value>true</value>  </property>  <property>  <name>mapreduce.cluster.administrators</name>  <value>yarn,hue <your_admin_group>s</value>  </property>  <property>  <name>mapreduce.job.acl-view-job</name>  <value>yarn,hue <your_admin_group></value>  </property>    Please try to add hue user and try executing your job ... View more

Please review the below Email discussion thread as well !   https://groups.google.com/a/cloudera.org/forum/#!topic/hue-user/fcRM3YiqAAA ... View more

Hello @upasana   Thanks for posting your query!   I see you are facing issue while calling the Livy's rest API for submitting the job and it is complaining on "j ava.lang.NoClassDefFoundError"   I would like to clarify, the error message "java.lang.NoClassDefFoundError" whilch you are getting is from the application (which holds the REST API call) or from the actual user job ?   If you are getting this error from the user application (which you are invoking through REST API) you might need to add the jar in executors/Driver's classpath   If you are getting above error in your application (which invokes the REST call) you might need to add the jar in your application's classpath   Please correct me if my above understanding is wrong  ... View more

Thanks for your input   Please check and share below configuration values from your Cloudera Manager's YARN configurations    yarn.admin.acl  mapreduce.job.acl-view-job ... View more

@mchia Is your cluster is Kerberos enabled ? And you have configured YARN ACLs on your cluster ?   Please check the below from your Cloudera Manager >> YARN configurations   a) Search for " Admin Users Applications List Visibility Settings" >> It should be selected as "All" b) Also you can check "Non-Admin Users Applications List Visibility Settings"  as well ... View more

Hello @tuckerrogerd @mchia   Thanks for posting your query!   Do you login as admin user in your Cloudera Cluster ? Also are you able to see the jobs in Resource Manager Web UI or Job History server ?     ... View more

Hello @Alealferez   Thanks for posting your query   Just wanted to double confirm your LDAP is ssl enabled ? Does it configured properly with Cloudera Manager as mentioned in below URL   https://streamsets.com/documentation/datacollector/3.5.2/help/datacollector/UserGuide/Configuration/Authentication.html ... View more

Hello @ursash   Glad to know your issue got relsoved !   Yes, the API calls " GET /jobs/ failed" might be suffered because of your conflict   ... View more

Hello @srikanthk   Thanks for posting your query with us.   I see the below documentation URL might help you with recommendations for CDSW   https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_requirements_supported_versions.html#cdsw_requirements_supported_versions   Hope this helps ... View more