About yvora

yvora · ‎08-14-2017

@Lukas Müller, try below way to create dataframes for data.json import json import requests r = requests.get("http://api.luftdaten.info/static/v1/data.json") df = sqlContext.createDataFrame([json.loads(line) for line in r.iter_lines()]) Reference: https://stackoverflow.com/questions/32418829/using-pyspark-to-read-json-file-directly-from-a-website

yvora · ‎07-31-2017

@Maya Tydykov, below thread might help you. https://community.hortonworks.com/questions/23242/caused-by-comgoogleprotobufinvalidprotocolbufferex.html

yvora · ‎07-28-2017

@PeiHe Zhang, can you please check the value of yarn application classpath ? It should have all below paths added in this property. <property> <name>yarn.application.classpath</name> <value>/etc/hadoop/conf/,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value> </property>

yvora · ‎07-17-2017

@Mateusz Grabowski, You should enable Dynamic Resource Allocation in Spark to automatically increase/decrease executors of an app as per resource availability. You can choose to enable DRA in either Spark or Zeppelin . 1) Enable DRA for Spark2 as below. https://community.hortonworks.com/content/supportkb/49510/how-to-enable-dynamic-resource-allocation-in-spark.html 2) Enable DRA via Livy Interpreter. Run all spark notebooks via livy interpreters. https://zeppelin.apache.org/docs/0.6.1/interpreter/livy.html

yvora · ‎07-13-2017

@Sami Ahmad, It looks like the document used an older property name. You should look for dfs.datanode.data.dir property instead dfs.data.dirs. ( dfs.data.dirs property was renamed to be dfs.datanode.data.dir in Hadoop2)

yvora · ‎07-11-2017

@Jeff Stafford, you can change the value of spark_log_dir from /etc/spark2/conf/spark-env.sh. Restart the Spark services after making configuration change. export SPARK_LOG_DIR=/dev/null

yvora · ‎07-07-2017

@suyash soni, Zeppelin notebook can be exported via UI in json format only. Exporting Zeppelin notebook in R extension is not supported. http://fedulov.website/2015/10/16/export-apache-zeppelin-notebooks/

yvora · ‎07-05-2017

@Paramesh malla, Is testWrite.txt file present on HDFS while running test code second time ? If yes, please delete /hdfs_nfs/hdfs_data/sampledata/testWrite.txt and rerun. HDFS only supports append, so if you intend to append data after file creation, use 'a' option. with open(filename,'a')as f: f.write(text)

yvora · ‎06-27-2017

@dhieru singh, In this case, you will need to validate each service manually. Typically, smoke tests perform below checks. * Check whether service is up or not * If it has UI , check if UI page is accessible or not * Run a simple sanity use case ( like submit sleep job in case of Hadoop ) Example: Follow below doc to validate health of Hadoop services manually. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-upgrade/content/run-hadoop-tests-24.html

yvora · ‎06-26-2017

@dhieru singh, if this is Ambari installed cluster, you can run service checks as below to validate cluster state. https://community.hortonworks.com/articles/11852/ambari-api-run-all-service-checks-bulk.html

Online	Offline
Last Visited	‎10-25-2018 06:32 PM

Member Since	‎10-24-2015 06:41 PM
Last Visited	‎10-25-2018 06:32 PM
Posts	171
Kudos received	375

Cloudera Community

Re: yarn cache files does not have execute permiss...

Re: What is the use of zookeeper.out?

Re: how to know the reason for missing blocks

Re: Best way to monitor/move hadoop files through ...

Re: Limit in number of Yarn jobs

Re: Loading a JSON File from URL into a Spark Data...

Re: How to fix size limit error when working with ...

Re: Class org.apache.hadoop.yarn.client.RequestHed...

Re: Spark 2 interpreter runs only 3 containers

Re: can't find parameter dfs.data.dirs

Re: How to change Spark2 History Server log locati...

Re: How to download the Zeppelin notebook with .R ...

Re: Unable to over write file with cp command thro...

Re: Are there some standard test cases once the I ...

Re: Are there some standard test cases once the I ...