question Re: HDPCD Spark Certification in Support Questions

HDPCD Spark Certification

madhusudhanbabu — Tue, 01 Nov 2016 01:12:01 GMT

1. Which version of HDP sandbox is being used ?

2. Which version of Spark being used ?

3. What kind of IDE options are available during the exam for Python? Apart from pyspark-shell, is there any IDE available like IPython or Zeppelin ? Is there any IDE option available which have auto suggestion option and where we can submit jobs to cluster. Please advise.

4. I have read few posts in hortonworks community, that we may use Spark RDDs or Spark Dataframes for accomplishing the tasks? Please confirm.

5. What is the pass percentage on average ?

Re: HDPCD Spark Certification

madhusudhanbabu — Tue, 01 Nov 2016 01:23:45 GMT

@Qi Wang : Could you please help with the above queries.

Re: HDPCD Spark Certification

qiwang — Tue, 01 Nov 2016 01:33:47 GMT

the test environment is on AMS virtual. When I took the test, it was HDP2.3 and I am not sure what version is used now. you could use the current sandbox for your exercise. Spark is something later than 1.4, probably 1.5. But the knowledge covered are all basic RDD and dataframe that are not very much linked to newer versions. test environment has no IDE. You use either gedit or vi base on you preference. debug with spark-shell or pyspark

couple notes on the exam

1. know RDD and dataframe api well. Go through all the docs in the test web page.

2. know how to import and export RDD/dataframe from/to csv files.

3. there is no limit on how you finish the task, so choose the technical you are most familiar with either the API or Spark SQL

4. test environment is quite slow in response, so be patient with it and leave enough time for tasks.

Good luck taking the exam.

Re: HDPCD Spark Certification

madhusudhanbabu — Tue, 01 Nov 2016 02:32:48 GMT

@Qi Wang

Thank you for your prompt response. Could you please help with below queries.

Current sandbox version is HDP 2.5 and supported Spark version is 1.6.2.

1. In the sandbox which I have downloaded, only vi is available, there is no gedit. Do we need to install gedit ?

2. I have learn that Apache Spark documentation and Hortonworks Spark documentation is available during exam.

Apache Spark Documentation: https://spark.apache.org/docs . Is this the right link ?

Hortonworks Spark documentation: What is hortonworks spark documentation link ?

Thanks in advance.

Re: HDPCD Spark Certification

madhusudhanbabu — Tue, 01 Nov 2016 02:46:23 GMT

@Qi Wang

3. Is there anyway, to activate intellisense/auto completion work in HDP environment for spark in python. Either using vi/gedit or by using pyspark shell.

Thanks in advance.

Re: HDPCD Spark Certification

qiwang — Tue, 01 Nov 2016 08:42:49 GMT

the exam is setup on ubuntu with centOS VM as HDP. gedit is on ubuntu. I guess you could install gedit on your own environment but it is very easy to use, so no worry. If you really want to try the test environment, try use HDPCD practice exam, very similar.

http://hortonworks.com/wp-content/uploads/2015/02/HDPCD-PracticeExamGuide1.pdf

The document will be accessible during exam. it is the link you used for apache site http://spark.apache.org/docs For Hortonworks document, it is under http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/index.html Stick with Apache document as the exam is not really anything Hortonworks specific.

There is no way to change the exam environment. You have very limited permissions.

Re: HDPCD Spark Certification

madhusudhanbabu — Tue, 01 Nov 2016 21:28:12 GMT

@Qi Wang Thanks a lot..that really helps.

Re: HDPCD Spark Certification

madhusudhanbabu — Thu, 03 Nov 2016 23:52:25 GMT

@Qi Wang

In Spark 1.6.* version, RDD/dataframe have functions to write only to below formats

rdd.saveAsTextFile / saveAsSequenceFile

df.write.orc / json / parquet / text / saveAsTable

Query: I am sure we can not download other csv packages (i.e. databricks..etc) during the test. Is there any way to write the output file in csv format. Please advise.

Thanks in advance.

Re: HDPCD Spark Certification

qiwang — Thu, 03 Nov 2016 23:58:59 GMT

Yes and check my answer on another thread

https://community.hortonworks.com/questions/46772/how-to-save-dataframe-as-text-file.html#answer-46773

Re: HDPCD Spark Certification

madhusudhanbabu — Sun, 06 Nov 2016 11:49:58 GMT

@Qi Wang

Is it safe to assume that the Databricks package will be available during the test to read and write to csv files ?

pyspark --packages com.databricks:spark-csv_2.10:1.4.0
df.write.format("com.databricks.spark.csv").option("header","true").save("file.csv")

Re: HDPCD Spark Certification

qiwang — Sun, 06 Nov 2016 23:03:09 GMT

That I am not sure. I did not use that library.

Re: HDPCD Spark Certification

madhusudhanbabu — Mon, 07 Nov 2016 06:08:43 GMT

@Qi Wang

Thank you. Apologies for repeating the same question again.

Which library you have used to write to csv file. I am planning to take the exam in python. Please advise.

Re: HDPCD Spark Certification

qiwang — Thu, 10 Nov 2016 08:42:03 GMT

I won't assume the package is available. Better find a way to do that in python.

Re: HDPCD Spark Certification

hmsvigle — Thu, 30 Mar 2017 18:42:57 GMT

Hi Wang,

thanks for all the tips shared. Its really helpful.

Could you please tell me, how many tasks provided and how many we have to complete to clear the test.

Also it would be great help, if you could share the questions asked, if you remember.

As there is no practice test available for HDPCD-Spark, I dont have any clue about the pattern of tasks.

Thank you

Himansu

Re: HDPCD Spark Certification

himanishkopalle — Fri, 22 Dec 2017 23:17:17 GMT

@Qi Wang

Hello Sir,

Thanks for your input.

Just one doubt about the pattern. Please let us know how many questions would be asked in the exam?
and what would be the passing criteria?

Thank you.