Issue running Impala Shell Commands with Oozie

kit24 — Fri, 16 Sep 2022 12:47:58 GMT

Hi,

When I run Impala shell commands with Oozie workflow based on the steps recommended here:
http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-schedule-with-oozie-tutorial/td-p/23906
https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/How-to-Schedule-Impala-Jobs-with-Oozie/ta-p/31277/jump-to/first-unread-message

I am getting an error thrown:

Traceback (most recent call last):

File "/app/localstorage/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/bin/../lib/impala-shell/impala_shell.py", line 38, in

<module>

from impala_client import (ImpalaClient, DisconnectedException, QueryStateException,

File "/app/localstorage/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala-shell/lib/impala_client.py", line 20, in <

module>

import sasl

File "/app/localstorage/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala-shell/ext-py/sasl-0.1.1-py2.7-linux-x86_64

.egg/sasl/__init__.py", line 1, in <module>

 

File "/app/localstorage/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala-shell/ext-py/sasl-0.1.1-py2.7-linux-x86_64

.egg/sasl/saslwrapper.py", line 7, in <module>

File "/app/localstorage/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala-shell/ext-py/sasl-0.1.1-py2.7-linux-x86_64

.egg/_saslwrapper.py", line 7, in <module>

File "/app/localstorage/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala-shell/ext-py/sasl-0.1.1-py2.7-linux-x86_64

.egg/_saslwrapper.py", line 6, in __bootstrap__

ImportError: /tmp/impala-shell-python-egg-cache-subuser/sasl-0.1.1-py2.7-linux-x86_64.egg-tmp/_saslwrapper.so: fail

ed to map segment from shared object: Operation not permitted

We are using kerberos and this folder has 777 permission, but still throwing this error.

How do we resolve this?

Re: Issue running Impala Shell Commands with Oozie

kit24 — Thu, 01 Feb 2018 15:35:12 GMT

Problem Solved

The issue was that the script /bin/impala_shell is hardcoded by cloudera on our nodes, and so the PYTHON_EGG_CACHE always got redefined to /tmp

Below is a snippet of the code in /bin/impala_shell:

# We should set the EGG_CACHE to a per-user temporary location.
# This follows what hue does.
PYTHON_EGG_CACHE=/tmp/impala-shell-python-egg-cache-${USER}
if [ ! -d ${PYTHON_EGG_CACHE} ]; then
  mkdir ${PYTHON_EGG_CACHE}
fi

Additionally, /tmp was not exec mounted on all nodes.

In order to solve the issue we were having, we added the below code to the impala shell script that we want to run with oozie:

export PYTHON_EGG_CACHE=/app/bds
export link_folder=/tmp/impala-shell-python-egg-cache-$(whoami)
if ! [ -L $link_folder ]
then
    rm -Rf "$link_folder"
    ln -sfn ${PYTHON_EGG_CACHE}${link_folder} ${link_folder}
fi
mkdir -p ${PYTHON_EGG_CACHE}${link_folder}

This creates a new link dir for PYTHON_EGG_CACHE, on a shared folder, which can be accessed by all nodes.

question Issue running Impala Shell Commands with Oozie in Archives of Support Questions (Read Only)

Issue running Impala Shell Commands with Oozie

Re: Issue running Impala Shell Commands with Oozie