@kenny creed you are using regexp_replace in spark which gives you string datatype. In spark you have to use cast to convert it. Given below an example which might help in solving your problem: Hope it helps! val res = df.select($"id", $"date", unix_timestamp($"date", "yyyy/MM/dd HH:mm:ss").cast(TimestampType).as("timestamp"), current_timestamp(), current_date()) ... View more

@I1095 Check this blog. You have a detailed comparison between ORC and Parquet. Other than that there are very in terms of use case. But in the future most I believe most of the improvements are being developed based on ORC i believe. 1. Many of the performance improvements provided in the Stinger initiative are dependent on features of the ORC format including block level index for each column. This leads to potentially more efficient I/O allowing Hive to skip reading entire blocks of data if it determines predicate values are not present there. Also the Cost Based Optimizer has the ability to consider column level metadata present in ORC files in order to generate the most efficient graph. 2. ACID transactions are only possible when using ORC as the file format ... View more

Hi @Harjinder Brar concat('{', u.swid ,'}') will concatenate bracs with the value from u.swid. For Example if the value of u.swid is TEST then it will be converted to {TEST} which will be used to join with o.swid column. Hope it Helps!! ... View more