I took the HDPCD: Spark (Python) exam yesterday. I had to save my solution code in text files. The exam guide said that these could should 'run in the shell'. I was unsure, however, whether this refers to the Linux shell or the (Py-)Spark shell. In the first case, it is necessary to add initializations: #!/usr/bin/env python from pyspark import SparkContext, HiveContext sc = SparkContext() sqlContext = HiveContext(sc) Is it necessary to add these lines to every solution in the HDPCD: Spark exam? ... View more

I want to attempt the HDPCD: Spark exam soon and have two questions: I found this PDF that specifies the versions for HDP, Spark, Scala and Python. Is this information up to date? Are there any plans to switch to Spark 2.x in the near future? What operating system is used in the exam environment? ... View more