Suppose I have the following piece of code: val a = sc.textfile("path/to/file") val b = a.filter(<something..>).groupBy(<something..>) val c = b.filter(<something..>).groupBy(<something..>) val d = c.<some transform> val e = d.<some transform> val sum1 = e.reduce(<reduce func>) val sum2 = b.reduce(<reduce func>) Note that I have not used any cache/persist command. Since the RDD b is being used again in the last action, will Spark automatically cache it? Or will it be recalculated again from the dataset? Will the behaviour be the same, if I use DataFrame for the above steps? Lastly, at any point of time will the RDDs c or d exist? Or will Spark look ahead to check that they are not used in any actions, and consequently chain the transformations for c and d into b and directly calculate e? I am new to Spark and am trying to understand the basics. Regards, Anirban ... View more

I am working with the HDP2.3 Rev6 VM for a self paced course. I am getting the below errors for the same query, when using aliases. select sum(ordertotal), year(order_date) from orders group by year(order_date) this query works fine. But if I use aliases, it fails. Am I missing something? Regards, Anirban. ... View more

We can create the same table using one of the below two queries: I have seen that they both result in the same table. so how do they differ? and if they differ, when do I use one over the other? CREATE TABLE sample_table STORED AS AVRO TBLPROPERTIES('avro.schema.url' = '<some location>'); CREATE TABLE sample_table ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' WITH SERDEPROPERTIES ('avro.schema.url'='file:///tmp/schema.avsc') STORED as INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'; ... View more