After some research it seems that the problem is that java is missing the snappy native files
We moved the snappy native files (under the lib/hadoop/lib/native) from our hadoop server to the nifi server and tried everything we can find. we exported the LD_LIBRARY_PATH and the JAVA_LIBRARY_PATH the this directory, added the line java.arg.16=-Djava.library.path=/path/to/snappy/native/directory and even put the native files under the $JAVA_HOME/lib/native directory. nothing helped
We used lsof on out java (NiFi) process and found out that there is no reference the those files, but there is a reference to the /tmp/snappy-1.0.5-libsnappyjava.so, which might be relevant.