Problem Solved The issue was that the script /bin/impala_shell is hardcoded by cloudera on our nodes, and so the PYTHON_EGG_CACHE always got redefined to /tmp Below is a snippet of the code in /bin/impala_shell: # We should set the EGG_CACHE to a per-user temporary location.
# This follows what hue does.
PYTHON_EGG_CACHE=/tmp/impala-shell-python-egg-cache-${USER}
if [ ! -d ${PYTHON_EGG_CACHE} ]; then
mkdir ${PYTHON_EGG_CACHE}
fi Additionally, /tmp was not exec mounted on all nodes. In order to solve the issue we were having, we added the below code to the impala shell script that we want to run with oozie: export PYTHON_EGG_CACHE=/app/bds
export link_folder=/tmp/impala-shell-python-egg-cache-$(whoami)
if ! [ -L $link_folder ]
then
rm -Rf "$link_folder"
ln -sfn ${PYTHON_EGG_CACHE}${link_folder} ${link_folder}
fi
mkdir -p ${PYTHON_EGG_CACHE}${link_folder} This creates a new link dir for PYTHON_EGG_CACHE, on a shared folder, which can be accessed by all nodes.
... View more
