Compression not working due to tmp folder previliges

[#1]

Hi Everyone,

I have a problem whenever i am trying to store my data in a compressed format with pig, Sqoop, or Spark. I know the problem is with mounting our /tmp folder to nonexec and this causes for instance snappy to give me this error:

java.lang.IllegalArgumentException: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.2-fe4e30d0-e4a5-4b1a-ae31-fd1861117288-libsnappyjava.so: /tmp/snappy-1.1.2-fe4e30d0-e4a5-4b1a-ae31-fd1861117288-libsnappyjava.so: failed to map segment from shared object: Operation not permitted

The solutions that i found in the internet is that either mount the /tmp folder to exec which is not an option for me as the sysadmin won't allow it due to security concerns.The other option is to change the java opts execution path to some other paths instead of /tmp.

I have tried the following approach, but it didn't solve the problem.
add these lines to hadoop-env.sh and sqoop-env

export HADOOP_OPTS="$HADOOP_OPTS -Dorg.xerial.snappy.tempdir=/newpath"
export HADOOP_OPTS="$HADOOP_OPTS -Djava.io.tmpdir=/newpath"

I would appreciate if you guys have any other solutions that could solve the issue.

Thanks

[#2]

Hello Saad ,

Did you find a solution ?

[#3]

Hello,

I had the same problem with Hive & HiveOnTez, but instead of include theses parameters in environments variables, I had to include them in "java_opts" (hiveserver2_java_opts, hive.tez.java.opts, ...) services' configurations (via cloudera interfaces).
The result is the same, parameters are include as argument to java execution, but they are centralized in cloudera configuration.

Regards,

[#4]

@tpophillat Welcome to the Cloudera Community!

As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.