How to insert nano seconds in the TimestampType in Spark?

[#1]

Spark Version : 1.6.1 I have a text table in hive having `timestamp` datatype with nanoseconds precision. Hive Table Schema:

c_timestamp     timestamp   

Hive Table data:

00:00:00.000000001

23:59:59.999999999

But as per the docs, from Spark 1.5 Timestamps are now stored at a precision of 1us, rather than 1ns Sample code:

SparkConf conf = new SparkConf(true).setMaster("yarn-cluster").setAppName("SAMPLE_APP");

SparkContext sc = new SparkContext(conf);

HiveContext hc = new HiveContext(sc);

DataFrame df = hc.table("testdb.tbl1");

Data is truncated to microseconds. Is there any way to use nanoseconds here?

[#2 · Accepted solution]

@Devender Yadav, It seems that TimeStampType do not support nanoseconds yet.

https://issues.apache.org/jira/browse/SPARK-17914 is opened to track this issue.

You might want to use some other data type like String to store this data if you require nanoseconds data.