Member since
05-15-2018
114
Posts
8
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
817 | 06-02-2020 06:22 PM | |
9424 | 06-01-2020 09:06 PM | |
1952 | 01-15-2019 08:17 PM | |
3335 | 12-21-2018 05:32 AM | |
3313 | 12-16-2018 09:39 PM |
06-14-2020
07:27 PM
Hello @paraskar , Thank you for posting your query. Please refer the below posts which assist you on dr.who attack https://community.cloudera.com/t5/Support-Questions/What-is-Dr-who-user-100s-of-yarn-jobs-are-getting-triggered/td-p/68026 https://community.cloudera.com/t5/Support-Questions/Dr-who-virus-in-my-YARN-how-to-resolve-it-using-firewalld/m-p/297156/highlight/true#M218626
... View more
06-14-2020
07:16 PM
Hello @monorels , Thank you for posting the query. Check the total size of the files present on the event logging directory. possibly if you have huge sized files you would need to increase the spark history server's heap memory. Also check from the logs (spark history server logs) and see if you are getting any errors while spark history server replays the event logs from HDFS path. If there are no errors observed try to enable DEBUG level logging. Even still no errors logged, possibly you might need to check the number of event logs and the allocated heap memory usage.
... View more
06-04-2020
08:17 PM
Hello @deligentAl , Thanks for posting your query. Please refer the integration guide - which has the examples https://kudu.apache.org/docs/developing.html#_kudu_integration_with_spark
... View more
06-02-2020
06:49 PM
Hello @renzhongpei , From the log4j properties file I see you are trying to write the logs in local file path [ l og4j.appender.FILE.File=/home/rzpt/logs/spark.log ] Please note that, with the above log4j properties, the executors & Driver (in cluster mode) basically tries to write log files on above specified path on all the nodes where containers (executor) runs If your requirement is such, you would need to follow command like this (assuming the log4j.properties file in your local /tmp path on the node where you execute spark2-submit) spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain --files /tmp/log4j.properties --conf spark.driver.extraJavaOptions="-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties" --master yarn --deploy-mode cluster sgp-1.0.jar Note that in above command's "-Dlog4j.configuration=log4j.properties" you can use as it is (i.e) you don't need to give the explicit local path such as file:// . since the executor would automatically pickup the log4j.properties from the container localised path
... View more
06-02-2020
06:34 PM
Hello @Mondi , Ideally you would be able to see the logs from Job history server (JHS) web UI (usually Resource manager redirects) to Job History server URL when the application finishes and while you access the logs. Were you not able to see the logs from JHS UI ? Any error you are seeing ? Alternatively, you can also execute (from CLI) #yarn logs <options> command to review the logs (if log aggregation is enabled)
... View more
06-02-2020
06:22 PM
Hello @mike_bronson7 , Thank you for posting your query You can execute 'get' on the same zookeeper client shell for the znode you would be able to get the hostname Example: zookeeper-shell.sh zoo_server1:2181 <<< "ls /brokers/ids/1018" It returns output as follows (example - in my case) [zk: localhost:2181(CONNECTED) 5] get /brokers/ids/10 {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://simple01.cloudera.com:9092"],"jmx_port":9393,"host":"simple01.cloudera.com","timestamp":"1590512066422","port":9092,"version":4} cZxid = 0x1619b ctime = Tue May 26 09:54:26 PDT 2020 mZxid = 0x1619b mtime = Tue May 26 09:54:26 PDT 2020 pZxid = 0x1619b cversion = 0 dataVersion = 1 aclVersion = 0 ephemeralOwner = 0x1722ddb1e844d50 dataLength = 238 numChildren = 0 so, my brokerID 10 is mapped with the host: simple01.cloudera.com
... View more
06-02-2020
05:28 PM
Hello @Venkat451 , Thank you for posting your query. From the error message shared (as below) I see the executor failing while it is trying to attach itself with the consumer group, more specifically, it is getting Authorisation exception while attaching to the group. ERROR org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2) org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-executor-<groupID> If you have authorisation mechanisms such as (sentry, kafka ACLs, Ranger) enabled on your cluster please grant necessary permissions to the consumer group https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/kafka-acl-examples.html
... View more
06-02-2020
04:26 AM
Hello @zanteb , Thank you for posting your query. While you are using with spark-submit you would require to pass the files (jass & keytab) with --files option on spark-submit just like [1] https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-spark-applications/content/running_spark_streaming_jobs_on_a_kerberos-enabled_cluster.html While doing so, your JAAS and keytab file would be shipped to executors and Application master /Driver (incase of cluster mode) If your external client is not spark and it is just a standalone java code (example) then you can just go ahead with passing " -Djava.security.auth.login.config=jaas.conf"" while executing the code and file can reside on the same client node
... View more
06-01-2020
09:16 PM
Hello @KarthikVenky , Thank you for posting your query. How about trying from Different browser on your laptop ? Does it give any different error Please check in chrome's Developer tools [Control + Shift + J ] - to see if there are any errors reported internally while loading the page
... View more
06-01-2020
09:06 PM
Hie @mig_aguir , Thank you for posting your query Could you please try running the job with - -conf spark.unsafe.sorter.spill.read.ahead.enabled=false
... View more
05-22-2020
05:56 PM
Hello @Moutasem , Thank you for posting your query. From the Error messages posted, it looks the agent complains on ports as it detects the conflicting usage of port between 2 services. Do you have multiple Kafka services (Like different set of Kafka brokers) added and managed by same Cloudera Manager ?
... View more
05-22-2020
05:43 PM
Hello @Mondi , Thank you for posting your query. Spark History server would replay the logs as soon as it gets the files (eventlogs) in the configured HDFS path [ /user/spark/applicationHistory]. The replay operation just reads the Event logs from HDFS path and loads in to memory to make it available for rendering. In your case, you have already confirmed that the file is present on the HDFS event logging directory. As a next step, could you please review the Spark History server logs and check if the replay operation is happening? Also, there are chances that if the file/directory permissions of event logs are incorrect the replay operation would fail silently, In such scenarios, you might need to enable DEBUG level logs to review whats wrong with replay operations. Hope this helps.
... View more
05-22-2020
05:29 PM
Hello @kvinod , These " org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException)" Exceptions commonly indicates that the HDFS file which is being used is being deleted or moved. Do you have parallel jobs accessing the same path in same time ?
... View more
05-22-2020
05:22 PM
@sahithi1 You can restart the Namenode in Ambari by selecting the Namenode process >> And from the drop down >>click on restart. And could you please share any specific issue which you are facing while restarting ?
... View more
03-05-2020
05:46 AM
It looks like the Class "org.apache.spark.metrics.sink.MQTTSink" is missing on your classpath. Could you please check if you can add the corresponding Jar which holds the same class in your Driver, Executor classpath ?
... View more
03-04-2020
11:27 PM
Hello @lakshmipathy , Please refer the below thread, hope this would help https://community.cloudera.com/t5/Support-Questions/How-to-define-topic-retention-with-kafka/td-p/222671
... View more
03-04-2020
09:30 PM
Hello @ravisro , I don't think there would be a straight forward way for this, (i.e) we might need to perform some sort of data cleansing work prior feeding it to spark in my view. Possibly the inputs shared does contains new line characters (\n) which might make spark to confuse with the data and new lines. I did some sort of data cleansing, (i.e) removing newlines gave me below result inputDf . show ( false ) +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
|_c0 |_c1 |_c2 |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
|2020-02-23 11:15:39|"Hi Craig, Please approve the standard pricing. No further amendments made "Legal System."Justification -XXX is the sole owner in China Thank you."|Approved|
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+ while running the same code from spark.
... View more
07-03-2019
08:33 AM
Hello @Dylan- Is your CSV files are in Git and you are pulling into your project through CDSW ? How big your CSV files are?
... View more
07-03-2019
08:30 AM
Hello @RichardJG , Thanks for posting your query. Based on the documentation, at present Kafka connect is not supported https://www.cloudera.com/documentation/kafka/latest/topics/kafka_known_issues.html#xd_583c10bfdbd326ba-590cb1d1-149e9ca9886--6fcb__section_ens_4bf_55 Is it possible for you to post the complete exception?
... View more
07-03-2019
08:27 AM
Hello @RajeshBodolla It seems like the actual exception is as below Caused by: org.apache.kafka.common.security.kerberos.NoMatchingRule: No rules apply to RAJESH@KAFKA4.CORP, rules [DEFAULT] Could you please share the complete stack? it seems it is somewhere not able to resolve the principal
... View more
03-04-2019
09:41 PM
Hello @Sankar Thanks for posting your query, You can try launching R engine in your CDSW and try running your code. Also, could you please share us what was the error which you are receiving while launching the job and how you are running the job ?
... View more
03-04-2019
09:18 PM
Hello @desind From the Message, it looks the SSL enabled URL is giving connection timeout. Could you try running the curl with -vvv option and see what are the request responses ?
... View more
03-04-2019
09:12 PM
Hello @regeamor Thank you for posting the query with us. Basically when you enable Dynamic allocation it gracefully remove the Idle containers which were idle for (60s as default) But when you lower the value will remove the executors frequently depending upon the executor's usage (just like the tasks getting allocated to those executors and the amount of tasks required) https://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors Also, the above said behaviour is upto Spark (and yarn container pre-emption wont be aware of it)
... View more
02-05-2019
09:59 PM
1 Kudo
Hello @Alfred_Lo, Thank you for posting your query. I could see you are using Oozie job and the job which you are referring as "SUCCEEDED" is oozie launcher job. So, the action which was triggered from oozie launcher got failed (Example: Your MR action, yarn action, spark action etc., ) but the oozie launcher succeeded and hence you are seeing the status of oozie launcher job as "SUCCEEDED"
... View more
01-23-2019
05:29 AM
Hello @Sankar Thanks for posting your query with us! From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager Does your normal spark jobs from CDSW is running fine ?
... View more
01-23-2019
05:23 AM
Hello @yasmin, Thanks for reaching us ! As per your reply it says that user who has triggered job, his user id is not listed on that particular node ? Is my understanding correct? ==> Yes you're right ! And your query regarding job slowness, we should consider the factors you mentioned along with below messages as well ~~~ ExecutorLostFailure (executor 1480 exited caused by one of the running tasks) Reason: Stale executor after cluster manager re-registered. I had 438 failures on 869 taks, that is a huge rate, another part has 873 out of 1236 ~~~ Here it seems the executors are getting lost and as a results tasks were dying. Could you please check the yarn logs for the application (by using #yarn logs command in my previous reply) and see if there is any errors in executor logs. This will help us to see if there is any spcific reason for executor failures Are you running job in spark-client mode or cluster mode?
... View more
01-15-2019
08:17 PM
1 Kudo
Hello @Rashmi22 Thanks for posting your query. Basically, Kafka is not dependent on HDFS service and brokers uses its own disks for storing the kafka data. So, in Kafka side there wont be any issues. But make sure you dont have any interruption on other services like Zookeeper (critical dependent for Kafka)
... View more
01-15-2019
07:11 PM
Yeah, thats right. There is some issue while rendering the images on your browser. Could you please try checking in Javascript console on any of your browser while running the snippet you may refer below URL for opening Javascript consoles: https://webmasters.stackexchange.com/questions/8525/how-do-i-open-the-javascript-console-in-different-browsers
... View more
01-15-2019
07:08 PM
Hello @dpugazhe Thanks for posting your query. From the message "/opt/cloudera/parcels/CDSW-1.2.2.p1.216803/scripts/cdsw-pod-logs.sh: line 40: kubectl: command not found" It seems your installation is missing the necessary kubernetes package. Could you please check if there was any installation errors on the node which you specified ?
... View more
01-15-2019
07:01 PM
Hello @yasmin Thanks for posting your query. From the error message attached I see the Nodemanager node is complaining that the user ID is not present on the node. Please check if the username is present on the node (where AM is running) or if you are using AD/LDAP just make sure the node is able to resolve the particular username You can also check more on container logs, by running #yarn logs -applicationId <applicationID> -appOwner <username_who_triggered_job>
... View more