@testingsauce I am also facing this issue. Saved df in HIVE using saveAsTable but when i try to fetch results using hiveContext.sql(query), it doesn't return anything. BADLY stuck. Please help ... View more

Could be a data skew issue. Checkout if any partition has huge chunk of the data compared to the rest. https://github.com/adnanalvee/spark-assist/blob/master/spark-assist.scala From the link above, copy the function "partitionStats" and pass in your data as a dataframe. It will show the maximum, minimum and average amount of data across your partitions like below. +------+-----+------------------+ |MAX |MIN |AVERAGE | +------+-----+------------------+ |135695|87694|100338.61149653122| +------+-----+------------------+ ... View more

You can add compression when you write your data. This will speed up the saving because the size of the data will smaller. Also increase the number of partition ... View more

Thanks! I would be interested to learn more when you are ready to announce it. ... View more

IBM offers free courses in Scala and other languages, they are free. There are tests at the end of the course once successful you can earn badges and showcase them. https://cognitiveclass.ai/ ... View more

oh! that worked. Thanks a lot! ... View more

A quick hack would be to use scala "substring" http://alvinalexander.com/scala/scala-string-examples-collection-cheat-sheet So what you can do is write a UDF and run the "new_time" column through it and grab upto time stamp you want. For example if you want just "yyyy-MM-dd HH:MM" as seen when you run the "df.show", your sub string code will be new_time.substring(0,15) which will yield "2015-12-06 12:40" pseudo code def getDateTimeSplit = udf((new_time:String) => { val s = new_time.substring(0,15) return s }) ... View more

@Adnan Alvee that is impressive indeed, ORC has additional benefits you will see on the Hive side. Glad you found it of use. ... View more

We can use rank approach which is faster than max , max scans the table twice: Here , partition column is load_date: select ld_dt.txnno , ld_dt.txndate , ld_dt.custno , ld_dt.amount , ld_dt.productno , ld_dt.spendby , ld_dt.load_date from (select *,dense_rank() over (order by load_date desc) dt_rnk from datastore_s2.transactions)ld_dt where ld_dt.dt_rnk=1 ... View more