Support Questions

Find answers, ask questions, and share your expertise

Impala Catalog Server supervisor permissions

Explorer

I have a problem on a CDH 5.8.0 parcels testsystem starting the Impala Catalog Server. 

The error message is the following: 

 

Can't open /var/run/cloudera-scm-agent/process/148-impala-CATALOGSERVER/supervisor.conf: Permission denied.

The permissions for this file are: 

 

-rw------- 1 root   root    2970 Jun 27 23:01 supervisor.conf

All other files and directories in the /var/run/cloudera-scm-agent/process/148-impala-CATLOGSERVER folder having owner impala and group  impala. 

When I change the permissions it will be overwritten on the next start. 

What is wrong here? 

And what needs to be changed. I recently installed a different service and changed the visudo file, but I have not changed the Impala configuration.

 

Thanks for the advice.

1 ACCEPTED SOLUTION

Champion

since i see you logs says that you are  downlading the parcels from /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../..

 

it says ( in your log file ) go to

 

 /var/log/impalad/impalad.ERROR

it may direct to mini-dump logs (/var/log/impala-minidumps/impalad / )

 

if you open the file and if you come across the JRE runtime error plus if it complains about the core dump .

 

 

 

Then it is the bug that is related to Linux kernel , impala's JRE error .

the fix is to increase the Stack thread  . do the below steps

 

the error should look like this

 

A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fa6b9f80c18, pid=3819, tid=0x00007fa6cfdb4900
#
# JRE version:  (8.0_131-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

The fix is

In  Cloduera manager UI -> Cluster -> Go to Impala - Configuration tab .

 

Search 

 

Step 1 :

Impala Daemon properties:

Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)

apply the  JAVA_TOOL_OPTIONS=-Xss2m

 

Steo 2 :

 

Apply the same for Impala Catalog -

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)"

before you restart the catalog daemon.

JAVA_TOOL_OPTIONS=-Xss2m

 

Reason - The crash was caused by a change to the linux kernel that modified the memory layout around JAVAthread stacks.

 

JIRA -   - https://issues.apache.org/jira/browse/DAEMON-363

 

Please let me know if that works
 

View solution in original post

14 REPLIES 14

Champion

Would you mind deleting the Impala Catalog and Statestore instance . Add a new instance one more . see if those errors popping up.

i hardly  remeber I had this issue . the above fixed it for me .

 

New Contributor

I have the same problem. I installed CDH 5.11.1 on a single machine AWS EC2. When starting impala service, Impala Catalog Server and Impala Daemon don't start. I removed the role and added again but the problem still persists.

Impala Daemon stderr :

...

+ exec /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/154-impala-IMPALAD/impala-conf/impalad_flags
Redirecting stderr to /var/log/impalad/impalad.ERROR

...

Can't open /run/cloudera-scm-agent/process/154-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/154-impala-IMPALAD/proc.json: Permission denied

 

Impala Catalog Server stderr:

...

exec /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../../bin/catalogd --flagfile=/run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/impala-conf/catalogserver_flags
Redirecting stderr to /var/log/catalogd/catalogd.ERROR

...

Can't open /run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/proc.json: Permission denied

 

Any help will be appreciated.

 

Explorer

Deleting the instances and adding them again did not solve the problem. 

The issue still exists. 

Champion

since i see you logs says that you are  downlading the parcels from /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../..

 

it says ( in your log file ) go to

 

 /var/log/impalad/impalad.ERROR

it may direct to mini-dump logs (/var/log/impala-minidumps/impalad / )

 

if you open the file and if you come across the JRE runtime error plus if it complains about the core dump .

 

 

 

Then it is the bug that is related to Linux kernel , impala's JRE error .

the fix is to increase the Stack thread  . do the below steps

 

the error should look like this

 

A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fa6b9f80c18, pid=3819, tid=0x00007fa6cfdb4900
#
# JRE version:  (8.0_131-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

The fix is

In  Cloduera manager UI -> Cluster -> Go to Impala - Configuration tab .

 

Search 

 

Step 1 :

Impala Daemon properties:

Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)

apply the  JAVA_TOOL_OPTIONS=-Xss2m

 

Steo 2 :

 

Apply the same for Impala Catalog -

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)"

before you restart the catalog daemon.

JAVA_TOOL_OPTIONS=-Xss2m

 

Reason - The crash was caused by a change to the linux kernel that modified the memory layout around JAVAthread stacks.

 

JIRA -   - https://issues.apache.org/jira/browse/DAEMON-363

 

Please let me know if that works
 

Explorer

works, I will accept this as solution

Champion

sounds good . 

whats ur kernel / os version ? 

New Contributor

It solved the problem.

 

OS: Ubuntu 14.04.3  (on AWS)

kernel: 3.13.0-121-generic

 

Thank you for detail explanation.

Champion

You Welcome mate 🙂

New Contributor
i tried as u say butmy impala catalog server is not started my Cdh is 5.11 using ubutu 16.04 lts kernal version is 4.X generic

Champion

could u share the logs .  

what is the status of other impala dameons ? 

what version of jdk are you using ? 

 

 

 

 

New Contributor

Screenshot from 2017-07-06 18-18-18.pngScreenshot from 2017-07-06 18-24-06.png

 

yes my impala daemon is working after adding the java_tool_options=-Xss2m but as same to catalog server i add java_tool_options=-Xss2m its not starting 

 

and my java version is

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
 

New Contributor

Thank you for sharing the solution.

Champion

@rroyYou Welcome 🙂

Explorer

Great it helped me too. 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.