thanks. The application is in Java, so what is going wrong above in the join statement? ... View more

Thanks! Does it then write to the check point dir after parsing every message (i.e after processing each streaming RDD) ? Is it an HDFS path? Will the HDFS hit everytime slow down the process? Also, If there are many kafka consumer groups then will this create a separate checkpoint dir for each consumer group? ... View more

@Mark Wallace Can you periodically run uncacheTable("table name") followed by cachetable("table name")? Both are inherited by HiveContext from SQLContext. Put the code you have above in a method along with adding lines for uncache and then simply call the method every hour. I am sorry if I am missing something. ... View more

Thanks for the details, this is helpful ... View more