Thanks @Karthik Narayanan. I was able to resolve the issue. Before diving into the solutions, I should make the below statement - With NiFi 1.0 and 1.1, LZO compression cannot be achieved using the PutHDFS processor. The only supported compressions are the ones listed in the compression codec drop down. With the LZO related classes being present in the core-site.xml, the NiFi processor fails to run. The suggestion from the previous HCC post was to remove those classes. It needed to be retained so that NiFi's copy and HDP's copy of core-site are always in sync. NiFi 1.0 I created the hadoop-lzo jar by building it from sources and added the same to the NiFi lib directory and restarted NiFi. This resolved the issue and I am able to proceed using the PutHDFS without it erroring out. NiFi 1.1 Configure the processor's additional classpath to the jar file. No restart required. Note : This does not provide LZO compression, it just can run the processor without ERROR even when you have the LZO classes in the core site. UNSATISFIED LINK ERROR WITH SNAPPY I also had issue with Snappy Compression codec in NiFi. Was able to resolve it setting the path to the .so file. This did not work on the ambari-vagrant boxes, but I was able to get this working on an openstack cloud instance. The issue on the virtual box could be systemic. To resolve the link error, I copied the .so files from HDP cluster and recreated the links. And as @Karthik Narayanan suggested, added the java library path to the directory containing the .so files. Below is the list of .so and links And below is the bootstrap configuration change ... View more

Unfortunately not every property currently supports using the variable registry. The way you can tell is in the documentation for a processor, by looking at the property description. If it says "Supports Expression Language: true" then you can reference variables. For example, with PutHDFS it looks like only the Directory property currently supports it: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html ... View more

This is a known problem when Phoenix is enabled, see similar posts here: https://community.hortonworks.com/questions/57874/error-unable-to-find-orgapachehadoophbaseipccontro.html That class is actually from Phoenix: https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/hadoop/hbase/ipc/controller/ServerRpcControllerFactory.java It will be fixed in Apache NiFi 1.1 by allowing users to specify the path to the phoenix client JAR. For now you can copy phoenix-client.jar to nifi_home/work/nar/extensions/nifi-hbase_1_1_2-client-service-nar-1.1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ obviously adjusting the directories for your version. ... View more

There is no hard-coded limit, but there is definitely some limit in terms of performance. Flow File attributes are held in memory (in addition to being persisted to disk) so more attributes means more Java objects on the heap, means more garbage collection pressure. I don't think there is any way to set what the limit is because it depends on how much data is in the attributes, how much memory you have, how many flow files, etc. ... View more