https://community.cloudera.com/t5/Archives-of-Support-Questions/Morphline-ReadAvroParquetFile-timestamp-problem/m-p/23665#M4610 <P>Hi all, i'm using the following morphline to index some parquet files:</P> <P>&nbsp;</P> <PRE>morphlines : [ { id : morphline1 importCommands : ["org.kitesdk.**","org.apache.solr.**"] commands : [ { readAvroParquetFile { projectionSchemaString : """ { "name": "record_parquet", "namespace": "parquet.avro", "type": "record", "fields": [ { "name": "id", "type":["null", "string"] }, { "name": "date_time", "type":["null", "string" ]}, { "name": "sessionid", "type": ["null","string" ]}, { "name": "client_id", "type": ["null","string" ]} ] } """ # supportedMimeTypes : [avro/binary] # projectionSchemaString : "" # optional, avro json schema blurb for getSchema() # projectionSchemaFile : /path/to/syslog.avsc } } { extractAvroPaths { flatten : true paths : { id : /id date_time : "/date_time" session_id : /sessionid client_id : /client_id } } } { addValues { # add values "text/log" and "text/log2" to the source_type output field Channel : [canal] } } { logDebug { format : "output record: {}", args : ["@{}"] } } # load the record into a Solr server or MapReduce Reducer { loadSolr { solrLocator : { collection : parquet_test # Name of solr collection zkHost : "$IP:2181/solr" # ZooKeeper ensemble batchSize : 1000 # batchSize } } } ] } ]</PRE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Everything is going well except the fact that the tdate_time field (already in solr format, and date type in the schema.xml) is converted in unix epoch format. Any idea about ? Thanks in advance</P> Fri, 16 Sep 2022 09:19:04 GMT fkrantz 2022-09-16T09:19:04Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Morphline-ReadAvroParquetFile-timestamp-problem/m-p/23665#M4610 <P>Hi all, i'm using the following morphline to index some parquet files:</P> <P>&nbsp;</P> <PRE>morphlines : [ { id : morphline1 importCommands : ["org.kitesdk.**","org.apache.solr.**"] commands : [ { readAvroParquetFile { projectionSchemaString : """ { "name": "record_parquet", "namespace": "parquet.avro", "type": "record", "fields": [ { "name": "id", "type":["null", "string"] }, { "name": "date_time", "type":["null", "string" ]}, { "name": "sessionid", "type": ["null","string" ]}, { "name": "client_id", "type": ["null","string" ]} ] } """ # supportedMimeTypes : [avro/binary] # projectionSchemaString : "" # optional, avro json schema blurb for getSchema() # projectionSchemaFile : /path/to/syslog.avsc } } { extractAvroPaths { flatten : true paths : { id : /id date_time : "/date_time" session_id : /sessionid client_id : /client_id } } } { addValues { # add values "text/log" and "text/log2" to the source_type output field Channel : [canal] } } { logDebug { format : "output record: {}", args : ["@{}"] } } # load the record into a Solr server or MapReduce Reducer { loadSolr { solrLocator : { collection : parquet_test # Name of solr collection zkHost : "$IP:2181/solr" # ZooKeeper ensemble batchSize : 1000 # batchSize } } } ] } ]</PRE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Everything is going well except the fact that the tdate_time field (already in solr format, and date type in the schema.xml) is converted in unix epoch format. Any idea about ? Thanks in advance</P> Fri, 16 Sep 2022 09:19:04 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Morphline-ReadAvroParquetFile-timestamp-problem/m-p/23665#M4610 fkrantz 2022-09-16T09:19:04Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Morphline-ReadAvroParquetFile-timestamp-problem/m-p/23781#M4611 I resolved this issue.<BR />There is no need of projectionSchemaString in readerAvroparquetFile.<BR />So after removing it, everything was working.<BR /><BR />Cheers. Fri, 16 Jan 2015 14:04:28 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Morphline-ReadAvroParquetFile-timestamp-problem/m-p/23781#M4611 fkrantz 2015-01-16T14:04:28Z