@Qi Wang Thank you. Apologies for repeating the same question again. Which library you have used to write to csv file. I am planning to take the exam in python. Please advise. ... View more

@Qi Wang Is it safe to assume that the Databricks package will be available during the test to read and write to csv files ? pyspark --packages com.databricks:spark-csv_2.10:1.4.0 df.write.format("com.databricks.spark.csv").option("header","true").save("file.csv") ... View more

@Qi Wang In Spark 1.6.* version, RDD/dataframe have functions to write only to below formats rdd.saveAsTextFile / saveAsSequenceFile df.write.orc / json / parquet / text / saveAsTable Query: I am sure we can not download other csv packages (i.e. databricks..etc) during the test. Is there any way to write the output file in csv format. Please advise. Thanks in advance. ... View more

@Qi Wang Thanks a lot..that really helps. ... View more

@Qi Wang 3. Is there anyway, to activate intellisense/auto completion work in HDP environment for spark in python. Either using vi/gedit or by using pyspark shell. Thanks in advance. ... View more

@Qi Wang Thank you for your prompt response. Could you please help with below queries. Current sandbox version is HDP 2.5 and supported Spark version is 1.6.2. 1. In the sandbox which I have downloaded, only vi is available, there is no gedit. Do we need to install gedit ? 2. I have learn that Apache Spark documentation and Hortonworks Spark documentation is available during exam. Apache Spark Documentation: https://spark.apache.org/docs . Is this the right link ? Hortonworks Spark documentation: What is hortonworks spark documentation link ? Thanks in advance. ... View more

@Qi Wang : Could you please help with the above queries. ... View more