Hello @paraskar ,   Thank you for posting your query.   Please refer the below posts which assist you on dr.who attack   https://community.cloudera.com/t5/Support-Questions/What-is-Dr-who-user-100s-of-yarn-jobs-are-getting-triggered/td-p/68026 https://community.cloudera.com/t5/Support-Questions/Dr-who-virus-in-my-YARN-how-to-resolve-it-using-firewalld/m-p/297156/highlight/true#M218626     ... View more

Hello @monorels ,   Thank you for posting the query.   Check the total size of the files present on the event logging directory. possibly if you have huge sized files you would need to increase the spark history server's heap memory. Also check from the logs (spark history server logs) and see if you are getting any errors while spark history server replays the event logs from HDFS path. If there are no errors observed try to enable DEBUG level logging. Even still no errors logged, possibly you might need to check the number of event logs and the allocated heap memory usage. ... View more

Hello @deligentAl ,   Thanks for posting your query.   Please refer the integration guide - which has the examples    https://kudu.apache.org/docs/developing.html#_kudu_integration_with_spark ... View more

Hello @renzhongpei ,   From the log4j properties file I see you are trying to write the logs in local file path [ l og4j.appender.FILE.File=/home/rzpt/logs/spark.log ]    Please note that, with the above log4j properties,  the executors & Driver (in cluster mode) basically tries to write log files on above specified path on all the nodes where containers (executor) runs   If your requirement is such, you would need to follow command like this (assuming the log4j.properties file in your local /tmp path on the node where you execute spark2-submit)   spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain --files /tmp/log4j.properties --conf spark.driver.extraJavaOptions="-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties" --master yarn --deploy-mode cluster sgp-1.0.jar   Note that in above command's "-Dlog4j.configuration=log4j.properties" you can use as it is (i.e) you don't need to give the explicit local path such as file:// . since the executor would automatically pickup the log4j.properties from the container localised path ... View more

Hello @Mondi ,   Ideally you would be able to see the logs from Job history server (JHS) web UI (usually Resource manager redirects) to Job History server URL when the application finishes and while you access the logs. Were you not able to see the logs from JHS UI ? Any error you are seeing ?   Alternatively, you can also execute (from CLI) #yarn logs <options> command to review the logs (if log aggregation is enabled)  ... View more

Hello @Venkat451 ,   Thank you for posting your query.   From the error message shared (as below) I see the executor failing while it is trying to attach itself with the consumer group, more specifically, it is getting Authorisation exception while attaching to the group.   ERROR org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2) org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-executor-<groupID> If you have authorisation mechanisms such as (sentry, kafka ACLs, Ranger) enabled on your cluster please grant necessary permissions to the consumer group   https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/kafka-acl-examples.html ... View more

Hello @zanteb ,   Thank you for posting your query.   While you are using with spark-submit you would require to pass the files (jass & keytab) with --files option on spark-submit just like [1]   https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-spark-applications/content/running_spark_streaming_jobs_on_a_kerberos-enabled_cluster.html   While doing so, your JAAS and keytab file would be shipped to executors and Application master /Driver (incase of cluster mode)   If your external client is not spark and it is just a standalone java code (example) then you can just go ahead with passing " -Djava.security.auth.login.config=jaas.conf"" while executing the code and file can reside on the same client node ... View more

Hello @KarthikVenky ,   Thank you for posting your query.   How about trying from Different browser on your laptop ? Does it give any different error  Please check in chrome's Developer tools [Control + Shift + J  ]  - to see if there are any errors reported internally while loading the page ... View more

Hello @Moutasem ,   Thank you for posting your query.   From the Error messages posted, it looks the agent complains on ports as it detects the conflicting usage of port between 2 services. Do you have multiple Kafka services (Like different set of Kafka brokers) added and managed by same Cloudera Manager ? ... View more

Hello @Mondi ,   Thank you for posting your query.   Spark History server would replay the logs as soon as it gets the files (eventlogs) in the configured HDFS path [  /user/spark/applicationHistory]. The replay operation just reads the Event logs from HDFS path and loads in to memory to make it available for rendering.    In your case, you have already confirmed that the file is present on the HDFS event logging directory. As a next step, could you please review the Spark History server logs and check if the replay operation is happening?    Also, there are chances that if the file/directory permissions of event logs are incorrect the replay operation would fail silently, In such scenarios, you might need to enable DEBUG level logs to review whats wrong with replay operations.   Hope this helps. ... View more

Hello @kvinod ,   These " org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException)" Exceptions commonly indicates that the HDFS file which is being used is being deleted or moved.   Do you have parallel jobs accessing the same path in same time ? ... View more

@sahithi1 You can restart the Namenode in Ambari by selecting the Namenode process >> And from the drop down >>click on restart.   And could you please share any specific issue which you are facing while restarting ? ... View more

It looks like the Class  "org.apache.spark.metrics.sink.MQTTSink" is missing on your classpath.   Could you please check if you can add the corresponding Jar which holds the same class in your Driver, Executor classpath ? ... View more

Hello @lakshmipathy ,   Please refer the below thread, hope this would help   https://community.cloudera.com/t5/Support-Questions/How-to-define-topic-retention-with-kafka/td-p/222671 ... View more

Hello @ravisro ,   I don't think there would be a straight forward way for this, (i.e) we might need to perform some sort of data cleansing work prior feeding it to spark in my view. Possibly the inputs shared does contains new line characters (\n) which might make spark to confuse with the data and new lines.    I did some sort of data cleansing, (i.e) removing newlines gave me below result   inputDf . show ( false ) +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+ |_c0 |_c1 |_c2 | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+ |2020-02-23 11:15:39|"Hi Craig, Please approve the standard pricing. No further amendments made "Legal System."Justification -XXX is the sole owner in China Thank you."|Approved| +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+  while running the same code from spark. ... View more

Hello @Dylan-    Is your CSV files are in Git and you are pulling into your project through CDSW ? How big your CSV files are? ... View more

Hello @RichardJG ,   Thanks for posting your query.   Based on the documentation, at present Kafka connect is not supported   https://www.cloudera.com/documentation/kafka/latest/topics/kafka_known_issues.html#xd_583c10bfdbd326ba-590cb1d1-149e9ca9886--6fcb__section_ens_4bf_55   Is it possible for you to post the complete exception? ... View more

Hello @RajeshBodolla    It seems like the actual exception is as below   Caused by: org.apache.kafka.common.security.kerberos.NoMatchingRule: No rules apply to RAJESH@KAFKA4.CORP, rules [DEFAULT]   Could you please share the complete stack? it seems it is somewhere not able to resolve the principal ... View more

Hello @Sankar   Thanks for posting your query,   You can try launching R engine in your CDSW and try running your code.   Also, could you please share us what was the error which you are receiving while launching the job and how you are running the job ?   ... View more

Hello @desind   From the Message, it looks the SSL enabled URL is giving connection timeout.   Could you try running the curl with -vvv option and see what are the request responses ? ... View more

Hello @regeamor   Thank you for posting the query with us.   Basically when you enable Dynamic allocation it gracefully remove the Idle containers which were idle for (60s as default) But when you lower the value will remove the executors frequently depending upon the executor's usage (just like the tasks getting allocated to those executors and the amount of tasks required)   https://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors   Also, the above said behaviour is upto Spark (and yarn container pre-emption wont be aware of it) ... View more

Hello @Alfred_Lo,   Thank you for posting your query.   I could see you are using Oozie job and the job which you are referring as "SUCCEEDED" is oozie launcher job.    So, the action which was triggered from oozie launcher got failed (Example: Your MR action, yarn action, spark action etc., ) but the oozie launcher succeeded and hence you are seeing the status of oozie launcher job as "SUCCEEDED"     ... View more

Hello @Sankar   Thanks for posting your query with us!   From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager   Does your normal spark jobs from CDSW is running fine ? ... View more

Hello @yasmin,   Thanks for reaching us !   As per your reply it says that user who has triggered job, his user id is not listed on that particular node ? Is my understanding correct? ==> Yes you're right !   And your query regarding job slowness, we should consider the factors you mentioned along with below messages as well   ~~~ ExecutorLostFailure (executor 1480 exited caused by one of the running tasks) Reason: Stale executor after cluster manager re-registered.   I had 438 failures on 869 taks, that is a huge rate, another part has 873 out of 1236 ~~~   Here it seems the executors are getting lost and as a results tasks were dying.   Could you please check the yarn logs for the application (by using #yarn logs command in my previous reply) and see if there is any errors in executor logs. This will help us to see if there is any spcific reason for executor failures   Are you running  job in spark-client mode or cluster mode?   ... View more

Yeah, thats right. There is some issue while rendering the images on your browser. Could you please try checking in Javascript console on any of your browser while running the snippet   you may refer below URL for opening Javascript consoles:   https://webmasters.stackexchange.com/questions/8525/how-do-i-open-the-javascript-console-in-different-browsers ... View more

Hello @dpugazhe   Thanks for posting your query.   From the message   "/opt/cloudera/parcels/CDSW-1.2.2.p1.216803/scripts/cdsw-pod-logs.sh: line 40: kubectl: command not found"   It seems your installation is missing the necessary kubernetes package. Could you please check if there was any installation errors on the node which you specified ?     ... View more

Hello @yasmin   Thanks for posting your query.   From the error message attached I see the Nodemanager node is complaining that the user ID is not present on the node. Please check if the username is present on the node (where AM is running) or if you are using AD/LDAP just make sure the node is able to resolve the particular username    You can also check more on container logs, by running    #yarn logs -applicationId <applicationID> -appOwner <username_who_triggered_job>    ... View more