How to insert nano seconds in the TimestampType in Spark?

dev_emr — Mon, 27 Mar 2017 19:37:29 GMT

Spark Version : 1.6.1 I have a text table in hive having `timestamp` datatype with nanoseconds precision. Hive Table Schema:

c_timestamp     timestamp

Hive Table data:

00:00:00.000000001

23:59:59.999999999

But as per the docs, from Spark 1.5 Timestamps are now stored at a precision of 1us, rather than 1ns Sample code:

SparkConf conf = new SparkConf(true).setMaster("yarn-cluster").setAppName("SAMPLE_APP");

SparkContext sc = new SparkContext(conf);

HiveContext hc = new HiveContext(sc);

DataFrame df = hc.table("testdb.tbl1");

Data is truncated to microseconds. Is there any way to use nanoseconds here?

Re: How to insert nano seconds in the TimestampType in Spark?

yvora — Wed, 29 Mar 2017 04:23:18 GMT

@Devender Yadav, It seems that TimeStampType do not support nanoseconds yet.

You might want to use some other data type like String to store this data if you require nanoseconds data.