Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Catalog Server supervisor permissions

Solved Go to solution

Impala Catalog Server supervisor permissions

Explorer

I have a problem on a CDH 5.8.0 parcels testsystem starting the Impala Catalog Server. 

The error message is the following: 

 

Can't open /var/run/cloudera-scm-agent/process/148-impala-CATALOGSERVER/supervisor.conf: Permission denied.

The permissions for this file are: 

 

-rw------- 1 root   root    2970 Jun 27 23:01 supervisor.conf

All other files and directories in the /var/run/cloudera-scm-agent/process/148-impala-CATLOGSERVER folder having owner impala and group  impala. 

When I change the permissions it will be overwritten on the next start. 

What is wrong here? 

And what needs to be changed. I recently installed a different service and changed the visudo file, but I have not changed the Impala configuration.

 

Thanks for the advice.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Impala Catalog Server supervisor permissions

Champion

since i see you logs says that you are  downlading the parcels from /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../..

 

it says ( in your log file ) go to

 

 /var/log/impalad/impalad.ERROR

it may direct to mini-dump logs (/var/log/impala-minidumps/impalad / )

 

if you open the file and if you come across the JRE runtime error plus if it complains about the core dump .

 

 

 

Then it is the bug that is related to Linux kernel , impala's JRE error .

the fix is to increase the Stack thread  . do the below steps

 

the error should look like this

 

A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fa6b9f80c18, pid=3819, tid=0x00007fa6cfdb4900
#
# JRE version:  (8.0_131-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

The fix is

In  Cloduera manager UI -> Cluster -> Go to Impala - Configuration tab .

 

Search 

 

Step 1 :

Impala Daemon properties:

Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)

apply the  JAVA_TOOL_OPTIONS=-Xss2m

 

Steo 2 :

 

Apply the same for Impala Catalog -

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)"

before you restart the catalog daemon.

JAVA_TOOL_OPTIONS=-Xss2m

 

Reason - The crash was caused by a change to the linux kernel that modified the memory layout around JAVAthread stacks.

 

JIRA -   - https://issues.apache.org/jira/browse/DAEMON-363

 

Please let me know if that works
 

14 REPLIES 14

Re: Impala Catalog Server supervisor permissions

Champion

Would you mind deleting the Impala Catalog and Statestore instance . Add a new instance one more . see if those errors popping up.

i hardly  remeber I had this issue . the above fixed it for me .

 

Re: Impala Catalog Server supervisor permissions

New Contributor

I have the same problem. I installed CDH 5.11.1 on a single machine AWS EC2. When starting impala service, Impala Catalog Server and Impala Daemon don't start. I removed the role and added again but the problem still persists.

Impala Daemon stderr :

...

+ exec /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/154-impala-IMPALAD/impala-conf/impalad_flags
Redirecting stderr to /var/log/impalad/impalad.ERROR

...

Can't open /run/cloudera-scm-agent/process/154-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/154-impala-IMPALAD/proc.json: Permission denied

 

Impala Catalog Server stderr:

...

exec /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../../bin/catalogd --flagfile=/run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/impala-conf/catalogserver_flags
Redirecting stderr to /var/log/catalogd/catalogd.ERROR

...

Can't open /run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/proc.json: Permission denied

 

Any help will be appreciated.

 

Re: Impala Catalog Server supervisor permissions

Explorer

Deleting the instances and adding them again did not solve the problem. 

The issue still exists. 

Re: Impala Catalog Server supervisor permissions

Champion

since i see you logs says that you are  downlading the parcels from /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../..

 

it says ( in your log file ) go to

 

 /var/log/impalad/impalad.ERROR

it may direct to mini-dump logs (/var/log/impala-minidumps/impalad / )

 

if you open the file and if you come across the JRE runtime error plus if it complains about the core dump .

 

 

 

Then it is the bug that is related to Linux kernel , impala's JRE error .

the fix is to increase the Stack thread  . do the below steps

 

the error should look like this

 

A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fa6b9f80c18, pid=3819, tid=0x00007fa6cfdb4900
#
# JRE version:  (8.0_131-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

The fix is

In  Cloduera manager UI -> Cluster -> Go to Impala - Configuration tab .

 

Search 

 

Step 1 :

Impala Daemon properties:

Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)

apply the  JAVA_TOOL_OPTIONS=-Xss2m

 

Steo 2 :

 

Apply the same for Impala Catalog -

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)"

before you restart the catalog daemon.

JAVA_TOOL_OPTIONS=-Xss2m

 

Reason - The crash was caused by a change to the linux kernel that modified the memory layout around JAVAthread stacks.

 

JIRA -   - https://issues.apache.org/jira/browse/DAEMON-363

 

Please let me know if that works
 

Re: Impala Catalog Server supervisor permissions

Explorer

works, I will accept this as solution

Re: Impala Catalog Server supervisor permissions

Champion

sounds good . 

whats ur kernel / os version ? 

Re: Impala Catalog Server supervisor permissions

New Contributor

It solved the problem.

 

OS: Ubuntu 14.04.3  (on AWS)

kernel: 3.13.0-121-generic

 

Thank you for detail explanation.

Re: Impala Catalog Server supervisor permissions

Champion

You Welcome mate :)

Highlighted

Re: Impala Catalog Server supervisor permissions

New Contributor
i tried as u say butmy impala catalog server is not started my Cdh is 5.11 using ubutu 16.04 lts kernal version is 4.X generic