@Theyaa Matti, If you are running mapreduce job, You can update mapreduce.job.queuename to queue1 in mapred-site.xml This way all application will be launched to queue1 by default. ... View more

It worked. Thanks. ... View more

@Aditya Sirna, /tmp/testa dir is present. an livy user has permission to write to it. I received below output while trying to run webhdfs rest api. [root@xx user]# curl -i -X PUT "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?user.name=livy&op=CREATE" HTTP/1.1 307 TEMPORARY_REDIRECT Cache-Control: no-cache Expires: Tue, 26 Sep 2017 17:33:17 GMT Date: Tue, 26 Sep 2017 17:33:17 GMT Pragma: no-cache Expires: Tue, 26 Sep 2017 17:33:17 GMT Date: Tue, 26 Sep 2017 17:33:17 GMT Pragma: no-cache X-FRAME-OPTIONS: SAMEORIGIN Set-Cookie: hadoop.auth="u=livy&p=livy&t=simple&e=1506483197716&s=dRvADKPG0lrenLje4fmEEdgChFw="; Path=/; HttpOnly Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&user.name=livy&namenoderpcaddress=xxx:8020&createflag=&createparent=true&overwrite=false Content-Type: application/octet-stream Content-Length: 0 Server: Jetty(6.1.26.hwx) [root@xxx user]# curl -i -T /tmp/a.txt "http://<namenode host>:50070/webhdfs/v1/tmp/testa/a.txt?op=CREATE&overwrite=false" HTTP/1.1 100 Continue HTTP/1.1 307 TEMPORARY_REDIRECT Cache-Control: no-cache Expires: Tue, 26 Sep 2017 17:33:49 GMT Date: Tue, 26 Sep 2017 17:33:49 GMT Pragma: no-cache Expires: Tue, 26 Sep 2017 17:33:49 GMT Date: Tue, 26 Sep 2017 17:33:49 GMT xRAME-OPTIONS: SAMEORIGIN Location: http://xxx:50075/webhdfs/v1/tmp/testa/a.txt?op=CREATE&namenoderpcaddress=xx:8020&createflag=&createparent=true&overwrite=false Content-Type: application/octet-stream Content-Length: 0 Server: Jetty(6.1.26.hwx) ... View more

@Palash Dutta, Find below articles which shows how to rotate Hdfs logs and also zip the logs. https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.html https://community.hortonworks.com/questions/78699/how-to-rotate-and-archive-hdfs-audit-log-file.html ... View more

@Sebastien Chausson, you can refer to below document to set up spark keystore/truststore. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_spark-component-guide/content/spark-encryption.html ... View more

@suresh krishRefer to below thread, it returns Json response to get version for each component. https://community.hortonworks.com/questions/26581/rest-api-to-get-individual-hdp-component-version.html ... View more

kalyanasish chanda , This is a duplicate question. can you please remove this post ? ... View more

@kalyanasish chanda, After making config changes, you can directly run api to restart stale components. Find the post regarding how to restart stale components as below. https://community.hortonworks.com/articles/73272/how-to-find-and-fix-ambari-stale-configuration-usi.html ... View more

@parag dharmadhikari, the permission of /tmp dir is not correct on HDFS. Typically /tmp dir has 777 permission. Run below command on your cluster, it will resolve this permission denied error. hdfs dfs -chmod 777 /tmp ... View more

@Robert Hryniewicz, DSX local installation typically take 2.5 to 3 hours. Installation time mainly depends on network speed and available resources. Make sure your cluster meet the pre-requirements listed below. https://datascience.ibm.com/docs/content/local/requirements.html?linkInPage=true ... View more

@Lukas Müller, try below way to create dataframes for data.json import json import requests r = requests.get("http://api.luftdaten.info/static/v1/data.json") df = sqlContext.createDataFrame([json.loads(line) for line in r.iter_lines()]) Reference: https://stackoverflow.com/questions/32418829/using-pyspark-to-read-json-file-directly-from-a-website ... View more

@Theyaa Matti, If you want to move the logs to HTTPS, You will need to enable Wire-encryption in your cluster. Please follow below document to enable Wire encryption. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/ch_wire-webhdfs-mr-yarn.html ... View more

@Maya Tydykov, below thread might help you. https://community.hortonworks.com/questions/23242/caused-by-comgoogleprotobufinvalidprotocolbufferex.html ... View more

@Fahad Sarwar, Capacity scheduler does not have placement rules, where you can configure that if userX is running job, place the job in QueueX. Capacity scheduler ACLS only check whether the application is allowed to run in specific queue or not. In order to be able to run the job in a specific queue, you will need to set the queue config while running apps. ( If the queue config is not set, by default application gets launched in "default" queue) For mapreduce jobs : set -Dmapred.job.queue.name=<queue-name> or -Dmapred.job.queuename=<queue-name> yarn jar /usr/lib/gphd/hadoop-mapreduce/hadoop-mapreduce-examples-x.x.x-alpha-gphd-x.x.x.x.jar wordcount -D mapreduce.job.queuename=<queue-name> /tmp/test_input /user/fail_user/test_output For spark jobs: set --queue <queue-name> spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --queue <queue-name> /usr/hdp/2.x.x.x-xxxx/spark/lib/spark-examples-x.x.x.x.x.x.x-xxxx-hadoopx.x.x.x.x.x.x-xxxx.jar 10 ... View more

@PeiHe Zhang, can you please check the value of yarn application classpath ? It should have all below paths added in this property. <property> <name>yarn.application.classpath</name> <value>/etc/hadoop/conf/,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value> </property> ... View more

@Ivan Majnaric, It looks like your job is trying to find jar on local filesystem instead HDFS. java.io.FileNotFoundException: File file:/spark-examples_2.11-2.1.0.2.6.0.3-8.jar does not exist Please follow below article to set up Spark-Oozie Action workflow. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_spark-component-guide/content/ch_oozie-spark-action.html ... View more

@Jeff Stafford, you can change the value of spark_log_dir from /etc/spark2/conf/spark-env.sh. Restart the Spark services after making configuration change. export SPARK_LOG_DIR=/dev/null ... View more

@Tom Shiels, Zeppelin currently does not have this feature. It is definitely a good feature request. Feel free to create Apache Zeppelin Jira to include this feature in Zeppelin. ... View more

@Paramesh malla, Is testWrite.txt file present on HDFS while running test code second time ? If yes, please delete /hdfs_nfs/hdfs_data/sampledata/testWrite.txt and rerun. HDFS only supports append, so if you intend to append data after file creation, use 'a' option. with open(filename,'a')as f: f.write(text) ... View more

@John Ross Tasipit, can you please check Timeline server logs to check why is it going down ? ... View more

@dhieru singh, In this case, you will need to validate each service manually. Typically, smoke tests perform below checks. * Check whether service is up or not * If it has UI , check if UI page is accessible or not * Run a simple sanity use case ( like submit sleep job in case of Hadoop ) Example: Follow below doc to validate health of Hadoop services manually. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-upgrade/content/run-hadoop-tests-24.html ... View more

@Colton Rodgers, Good to know that it helped.Thanks for accepting answer. ... View more

@Colton Rodgers, can you please share livy interpreter logs ? ... View more