CSV to AVRO Conversion with NiFi Debugging, Checking Schemas

[#1]

I ran the same flow myself and examined the AVRO file in HDFS using AVRO Cli.

Even though I didn't specify SNAPPY compression, it was there in the file.

java -jar avro-tools-1.8.0.jar getmeta 23568764174290.avro

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
avro.codec snappyavro.schema {"type":"record","name":"people","doc":"Schema generated by Kite","fields":[{"name":"id","type":"long","doc":"Type inferred from '2'"},{"name":"first_name","type":"string","doc":"Type inferred from 'Gregory'"},{"name":"last_name","type":"string","doc":"Type inferred from 'Vasquez'"},{"name":"email","type":"string","doc":"Type inferred from '[email protected]'"},{"name":"gender","type":"string","doc":"Type inferred from 'Male'"},{"name":"ip_address","type":"string","doc":"Type inferred from '32.8.254.252'"},{"name":"company_name","type":"string","doc":"Type inferred from 'Janyx'"},{"name":"domain_name","type":"string","doc":"Type inferred from 'free.fr'"},{"name":"file_name","type":"string","doc":"Type inferred from 'NonMauris.xls'"},{"name":"mac_address","type":"string","doc":"Type inferred from '03-FB-66-0F-20-A3'"},{"name":"user_agent","type":"string","doc":"Type inferred from '\"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7;'"},{"name":"lat","type":"string","doc":"Type inferred from ' like Gecko) Version/5.0.4 Safari/533.20.27\"'"},{"name":"long","type":"double","doc":"Type inferred from '26.98829'"}]}

It's hard coded in NIFI.

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/sr...

It always adds SnappyCompression to every AVRO file. No options.

writer.setCodec(CodecFactory.snappyCodec());

Make sure you have a schema set:

Record schema

Record Schema:   ${inferred.avro.schema}

If you can make everything Strings and convert to other types later, you will be happier.

References:

https://www.linkedin.com/pulse/converting-csv-avro-apache-nifi-jeremy-dyer

https://community.hortonworks.com/questions/44063/nifi-avro-to-csv-or-json-to-csvnifi-convert-avro-t...

https://community.hortonworks.com/articles/28341/converting-csv-to-avro-with-apache-nifi.html