@Bala Vignesh N V , Thanks. In case, I am running same hql queries daily, then I need to store the result in the file . once hive tasks are completed then read the file and update the table ? and any other way to achieve it. ... View more

Hi @Bala Vignesh N V , looking for detailed info, assume below are the query in hql , select count(*) from table1; select sum(col1) from table1 group by col2 ; select sum(col3) from table1 group by col4 first query will returns single column which has count , remaining returns 10 columns each. where this result will store and how to process and more importantly what is the recommended way handle this or how industry handling this ? Thanks. ... View more

Hi All, I have given 8GB ram, but still it takes so much of time to load. pls find the attached screen shot. VM : Vmware HDP : hdp2.6 ... View more

@Namit Maheshwari Yes, there is a pattern for creating partition (yyyy-mm-dd) .. OK,your idea is , run the command and store the result and check for the existence of the partition ?? Is there any other simple way to check ? ... View more

Thanks for your answer, I am looking for kind of API to tell that, partition added or not.. I will be difficult to check the partition status daily and take call when there is a failure. ( Want to automate rather checking it manually) ... View more

@Vedant Jain , @Bernhard Walter well, may be my question is misleading you , let me elaborate it. val textFile = sc.textFile("hdfs://...") val counts = textFile.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...") a simple wordcount problem.. this piece of code given to driver program, which creates DAG and stages, and given task to respective worker nodes where actual operation is happening. Now, lets look at the first line of the program. From the file, RDD is generated (SparkContext implemented textFile() function which generates RDD from file). file is resides in worker node. from worker node, we needs to get the RDD out. In order to acheive that , Worker node ( or executor ) needs to have the SparkContext, Isn't it ? My Question is, How does executor gets the spark context ? ... View more