@ashok.kumar, same thing : Path does not exist: hdfs://cch1wpsteris01:8020/user/root/rdd; Create it ("/user/root/rdd") and grant proper permissions to it. ... View more

Hi @Vasanth Reddy, If I understood you correctly, I think you should be checking this: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setting_up_queues These are per container properties: yarn.scheduler.minimum-allocation-mb = 1024 yarn.scheduler.maximum-allocation-mb = 4096 yarn.schedular.minimum-allocation-vcores = 3 yarn.schedular.maximum-allocation-vcores = 3 While you're watching at cluster wide metrics: running containers to 72 Allocated Cpu v-cores to 72 Allocated memory MB to 120034 With the above settings, at one point you may have the same 72 containers with Allocated memory MB to 294912 (72 containers * maximum mb 4096) Let me know if I missunderstood your question. BR, David Bompart ... View more

Hi @ashok.kumar, The log is pointing to `java.io.FileNotFoundException: File does not exist: hdfs:/spark2-history`, meaning that in your spark-defaults.conf file, you have specified this directory to be your Spark Events logging dir. In this HDFS path, Spark will try to write it's event logs - not to be confused with YARN application logs, or your application logs -, and it's failing to find it. You might want to check your spark-defaults.conf file, and point `spark.eventLog.dir` to either a valid hdfs path, or a local path where your Spark Application has access to write. For example, assuming that your client is a Linux/MacOSX machine, you can simply create a /tmp/spark-events directory, grant appropriate write access to it, and then configure spark-defaults.conf to be like: spark.eventLog.dir=file:///tmp/spark-events This property can also be overriden, which will be easier for quick tests, i.e.: /spark-submit --class org.com.st.com.st.Iot_kafka_Consumer --master local[*] --conf spark.eventLog.dir="file:///tmp/spark-events" /usr/local/src/softwares/com.st-0.0.1-SNAPSHOT-jar-with-dependencies.jar BR, David Bompart ... View more

@Berry Österlund , SparkR is indeed no longer supported to access Hive, R is not considered in the HWC connector. ... View more

@Debjyoti Das have you tried replacing: spark = org.apache.spark.sql.SparkSession.builder.appName("MyApp")//.enableHiveSupport().getOrCreate; With: spark = org.apache.spark.sql.SparkSession.builder.appName("MyApp").getOrCreate(); ... View more

Hi @Sangram Gaikwad, They have been resolved and do not exist in HDP 3.0.1. They're just probably just missing on the Known Issues notes. ... View more

Spark-submit looks fine, this issue will take more than a forum to resolve, would require code and logs analysis I'd say. Meanwhile, I can only suggest to pass "-Dsun.security.krb5.debug=true" to the extraJavaOptions, and it would also help if you can set the following in log4j.properties file "log4j.logger.org.apache.spark.deploy.yarn.Client=DEBUG", then restart the application, hoping it will print more pointers. Also, if your KDC is an MIT KDC, double check that your principal has not set a 'Maximum Renewal Time' of 00:00:00 as explained here Another property to try out, depending on your application use case that may help is to set: --conf mapreduce.job.complete.cancel.delegation.tokens=false ... View more

Hi @Nikhil Raina, In simple words, a speculative execution means that Hadoop in overall doesn't try to fix slow tasks as it is hard to detect the reason (misconfiguration, hardware issues, etc), instead, it just launches another parallel/backup task for each task that is performing slower than the expected, on faster nodes. So these backup tasks are called speculative tasks and it can be enabled/disabled as its benefits are per use case and up to the Hadoop Admin to consider it to be beneficial or not; speculative execution has an impact on the cluster throughput and resource usage. You can find this in MapReduce or Spark for example. Hope it helps, David ... View more

@Daniel Müller You might want to increase the --executor-memory (and probably yarn.scheduler.maximum-allocation-mb as well) to a value that can hold your data size in memory. In some cases repartitioning is a better option. ... View more

Hi @Michael Bronson, Is it deleting everything else but the .inprogress files? The following is already present and fixed on HDP 2.6.4: https://issues.apache.org/jira/browse/SPARK-8617 Where one of the proposed changes was to use loading time for inprogress files as lastUpdated and keep using the modTime for completed files. First one will prevent deletion of inprogress job files. The second one will ensure that lastUpdated time won't change for completed jobs in an event of HistoryServer reboot. - Can you double check the .inprogress files timestamp. - Check they do not correspond to actual running applications (streaming apps for example) - Check permission on these files, and perhaps try to manually delete one of these lingering .inprogress files logged in as the spark user and see if it lets you remove one of them. - Restart the SHS and check the log to see if it prints any errors while trying to remove these .inprogress files. Similar error messages like: case t: Exception => logError("Exception in cleaning logs", t) logError(s"IOException in cleaning ${attempt.logPath}", t) logInfo(s"No permission to delete ${attempt.logPath}, ignoring.") Regards, David ... View more

Can you share (masked) the spark submit command and the full "delegation token has expired" stacktrace? Also what is the use case of your app? ... View more

Hi Mani, you might also want to increase the number of executors then, and may probably be able to lower the memory size. Try with: spark-submit --master yarn --deploy-mode client --driver-memory 5g --num-executors 6 --executor-memory 8g myclass myjar.jar param1 param1 param3 param4 param5 Tunning this requires lots of other information like input data size, application use case, datasource information, cluster resources available, etc. Keep tunning --num-executors --executor-memory and --executor-cores (5 is usually a good number) ... View more

Hi Mani, use - - executor-memory 10g instead of 6g, and remove the overHead config property. ... View more

Sure, can you share your spark-submit command with the arguments as well? Mask any sensitive information please. ... View more

Hi @Bharat Bhushan, It seems like there's something missconfigured in your spark-shell or spark-submit commands, which prevents a serviceRegistry creation and remains null: https://github.com/apache/hive/blob/master/llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java#L361 Please refer to: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_configure_a_spark_hive_connection.html Verify both spark.sql.hive.hiveserver2.jdbc.url and spark.hadoop.hive.llap.daemon.service.hosts are correct. ... View more

Hi @Aakriti Batra, The problem seems to be in the JAAS file passed to the executor, it would help to see it's content, but I'd rather suggest you to read this whole article instead: https://community.hortonworks.com/articles/56704/secure-kafka-java-producer-with-kerberos.html ... View more

hi @Ali, You might want to add "--keytab /path/to/the/headless-keytab", "--principal principalNameAsPerTheKeytab" and "--conf spark.hadoop.fs.hdfs.impl.disable.cache=true" to the spark-submit command. ... View more

Hi Mani, Consider boosting spark.yarn.executor.memoryOverhead from 6.6 GB to something higher than 8.2 GB, by adding "--conf spark.yarn.executor.memoryOverhead=10GB" to the spark-submit command. You could also workaround this by increasing the number of partitions (repartitioning) and number of executors. ... View more

Hi Kumar, It could be the case that you have a stale SHS process, so work on stopping any SHS process as PID 39259 and/or remove "rm" the SHS process id file usually under /var/run/spark2, and the restart it again. ... View more

With the same user used to run this application, what is the output of running in all and each of the nodemanager nodes "which java"? ProcessBuilder doesn't use the locations in environment variables, it looks for "java" in "/usr/bin/java", is that the java binary you're working on giving permissions? ... View more

Hi @Alexander Schätzle , Can you share the details on how are you setting these two properties? What is your spark version? Can you grep the line in your application log like "YarnAllocator": INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 10692 MB memory (including 2500 MB of overhead) That's what I got from Spark 1.6 (spark.yarn.executor.memoryOverhead=2500) and Spark 2.x (spark.executor.memoryOverhead=2500m). I don't think we can even do "2.5g" right? I didn't try programmatically though. In spark1.6.x: - spark.executor.memoryOverhead=2.5g, would not be picked up and you'll end up with default 384Mb. - spark.yarn.executor.memoryOverhead=2.5g, you would get an exception like 'Exception in thread "main" java.lang.NumberFormatException: For input string: "2.5g"' In spark2.x: - conf spark.yarn.executor.memoryOverhead=2.5g (deprecated warning), you would get an exception like "Exception in thread "main" java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Fractional values are not supported. Input was: 2.5" - conf spark.executor.memoryOverhead=2.5g (deprecated warning), you would get an exception like "Exception in thread "main" java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Fractional values are not supported. Input was: 2.5" ... View more

@Ahshan MD, @Kyunam Kim Configure HTTP authentication for HDFS and restart. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/_configuring_http_authentication_for_HDFS_YARN_MapReduce2_HBase_Oozie_Falcon_and_Storm.html ... View more

@Ramesh Prasad It can't be done via any regular/standard and automated way, also any other procedure is not recommended. So I think the interviewer just wanted to make sure you understand the basic MR/Hadoop concepts. ... View more

Hi, I dont see where is this being used after, nor understand what is it for? would anyone elaborate more on this line? #getent shadow |tail –n 3 >/root/ldap/passwords Thanks! Regards. ... View more