Can you retry all these tests and during the second cat, instead of "cat 'record02' ", cat something longer like "cat 'record123456789'". I'd like to see if tracking the file size is the issue, because record01 and record02 would be the same file size. ... View more

This most likely means there is another JAR you need to add... If you look at the pom file for the hadoop-azure JAR: http://central.maven.org/maven2/org/apache/hadoop/hadoop-azure/2.7.0/hadoop-azure-2.7.0.pom You can see all the dependencies it needs: <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-core</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-storage</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <scope>compile</scope> </dependency> My guess would be the azure-storage JAR is missing. This becomes a slippery slope though, because then azure-storage might have transitive dependencies as well. ... View more

I'm not totally sure if this is the problem, but given that NiFi has NARs with isolated class loading, adding something to the classpath usually isn't as simple as dropping it in the lib directory. The hadoop libraries NAR would be unpacked to this location: work/nar/extensions/nifi-hadoop-libraries-nar-<VERSION>.nar-unpacked/META-INF/bundled-dependencies/ You could trying putting the hadoop-azure.jar there, keeping in mind that if the work directory was removed, NiFi would unpack the original NAR again without your added jar. Some have had success creating a custom version of the hadoop libraries NAR to switch to other libraries: https://github.com/bbukacek/nifi-hadoop-libraries-bundle Right now Apache NiFi is based on Apache Hadoop 2.6.2. ... View more

I'm not 100% sure how LZO works, but in a lot of cases the codec ends up needing a native library. On a unix system you would set LD_LIBRARY_PATH to include the location of the .so files for the LZO codec, or put them in JAVA_HOME/jre/lib native directory. You could do something like: export LD_LIBRARY_PATH=/usr/hdp/2.2.0.0-1084/hadoop/lib/native bin/nifi.sh start That should let PutHDFS know about the appropriate libraries. ... View more

Can you try what Matt suggested above, to remove the "io.compression.codecs" from core-site.xml? I agree with him that this is likely related to the compression codecs, you can see in the stacktracke, the relevant lines are: org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2058) ~[na:na] at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128) ~[na:na] at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175) ~[na:na] at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getCompressionCodec(AbstractHadoopProcessor.java:375) ~[na:na] at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:220) ~[na:na] at ... View more

@bschofield Another idea for transferring large files over a high-latency network, might be the following... On the sending side use a SegmentContent processor to break a large FlowFile into many smaller segments, followed by a PostHTTP processor with the Concurrent Tasks increased higher than 1. This lets the sending side better utilize the network by concurrently sending segments. On the receiving side, use a ListenHTTP processor to received the segmented FlowFiles, followed by a MergeContent processor with a Merge Strategy of Defragment. The Defragment mode will merge all the segments back together to recreate the original FlowFile. ... View more

What version of NiFi is this? ... View more