Will you be using Hive to consume the data? ... View more

In theory hive streaming should run as fast as you can write to HDFS with the overhead happening when committing transactions. So to improve performance reduce the amount of transactions happening against a table. Here are a few knobs to turn. 1) Check the NiFi version. There was a bug fix in NiFi 1.2/ HDF 3.0. The processor has a new config property to set records per transaction. 2) Set the number of records per transaction high to improve throughput. 3) Increase the number of transactions per batch but not too high. If the data stream does not have enough data to quickly use the transactions in the batch they will be created and time out needlessly. As you are streaming use the show transactions command to monitor if transactions are timing out. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowTransactions 4) On the hive side increase the number of threads doing the compaction. hive.compactor.worker.threads Data is streamed into new files and hive does minor and major compactions to merge the new data into the existing ORC files. ... View more

1 & 2) Trying running "analyze table" to generate row and data size statistics. https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables%E2%80%93ANALYZE 3) ORC files are compressed with zlib by default. zlib offers a higher level of compression than snappy. If you don't want compression you have to set orc.compress to "NONE" 4) I believe this is referencing the hive compression feature. Text files can be gzipped or bzipped and still read by Hive. https://cwiki.apache.org/confluence/display/Hive/CompressedStorage ... View more

I should add my odbc.ini and odbcinst.ini files are under /Library/ODBC too. I am using System DSNs only. ... View more

Two things I would validate. 1) Make sure java.library.path has the mqjbnd in the path. I believe System.loadLibrary uses java.library.path instead of the LD_LIBRARY_PATH. Not sure about WMQ but if the newly loaded library needs to load another library then LD_LIBRARY_PATH will need to be set also. 2) Make sure the bit versions of the JVM and MQ client are the same. For example if you have a 64 bit JVM make sure it is loading the 64bit WMQ libraries and not 32 bit. ... View more

Thanks for the reply Andrew. I ended up using your alternative solution. The search and filter approach ended up being too tedious and couldn't guarantee accuracy. ... View more

The majority of commercial products used to move files (video in the media industry) use a UDP data channel with a TCP control channel to guarantee delivery and reduce the overhead of the TCP protocol. If a PutUDP was paired with ListenUDP could/should these processors be made to use a TCP control channel to group the UDP packets? Feels like we would be going beyond the intent of what a UDP processor should do. ... View more