Will you be using Hive to consume the data? ... View more

In theory hive streaming should run as fast as you can write to HDFS with the overhead happening when committing transactions. So to improve performance reduce the amount of transactions happening against a table. Here are a few knobs to turn. 1) Check the NiFi version. There was a bug fix in NiFi 1.2/ HDF 3.0. The processor has a new config property to set records per transaction. 2) Set the number of records per transaction high to improve throughput. 3) Increase the number of transactions per batch but not too high. If the data stream does not have enough data to quickly use the transactions in the batch they will be created and time out needlessly. As you are streaming use the show transactions command to monitor if transactions are timing out. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowTransactions 4) On the hive side increase the number of threads doing the compaction. hive.compactor.worker.threads Data is streamed into new files and hive does minor and major compactions to merge the new data into the existing ORC files. ... View more

1 & 2) Trying running "analyze table" to generate row and data size statistics. https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables%E2%80%93ANALYZE 3) ORC files are compressed with zlib by default. zlib offers a higher level of compression than snappy. If you don't want compression you have to set orc.compress to "NONE" 4) I believe this is referencing the hive compression feature. Text files can be gzipped or bzipped and still read by Hive. https://cwiki.apache.org/confluence/display/Hive/CompressedStorage ... View more