Best Practice - JSON to Avro, data type preserving

[#1]

Hi All, What is the best approach to convert a JSON to AVRO preserving the source datatypes. My source JSON has a field with timestamp (value would look like 2017-01-26T00:00:00-05:00) which I need to eventually insert to a hive table with column type timestamp. When I infer the schema, I get String for the timestamp field. Is there some pre-formatting that I can do on the timestamp field so that it gets inferred as timestamp field. Current flow is as below - JSON>>AVRO(infer/manually add schema)>>Streaming Insert to hive

[#2]

Can u please help on when we convert json to avro format how can we define logicaltype in avro

[#3]

While converting from json to avro format,how to get logicaltype in avro format.

And to get logicaltype in avro format,what we need to add in json data .

[#4]

https://community.hortonworks.com/questions/17996/nifi-applying-an-avro-schema-in-convertcsvtoavro.h...

[#5]

Here's an exmaple of what a Timestamp schema entry would look like: https://avro.apache.org/docs/1.8.0/spec.html#Timestamp+%28millisecond+precision%29

{ "type": "record",

"namespace": "test",

"name": "test1",

"fields": [

{ "name": "ID", "type": "string" },

{ "name": "V", "type": "float" },

{ "name": "T", "type":

{ "type": "long", "logicalType": "timestamp-millis" }

} ]

}

[#6]

@Arun A K

why not use an the new Record oriented processors, store the schema locally in nifi using the avroschemaregistry.

https://community.hortonworks.com/questions/113959/use-nifi-to-change-the-format-of-numeric-date-and...

[#7]

Thanks @Karthik Narayanan, I can use them now, but the question was when we had not released NiFi 1.2

[#8]

as per the avro doc, timestamp-millis is a logicaltype, so you have to use something like this...

{"name":"timestamp","type": {"type": "string", "logicalType": "timestamp-millis"}

see if that works. https://github.com/mtth/avsc/wiki/Advanced-usage#logical-types

[#9]

Thanks @Karthik Narayanan, same reason that I was on a NiFi that had Avro 1.7 the logicalType doesn't work. If I remember correctly, it threw an invalid avro schema error.

[#10]

hmm, strange which version of nifi were you on?

[#11]

@Karthik Narayanan it was NiFi 1.1 and looking at the pom.xml, I am assuming there is an avro 1.7 dependency. The logical time stamp stuff was introduced in avro 1.8