06-27-2017 02:13 PM - last edited on 11-02-2017 07:58 AM by cjervis
I have a problem on a CDH 5.8.0 parcels testsystem starting the Impala Catalog Server.
The error message is the following:
Can't open /var/run/cloudera-scm-agent/process/148-impala-CATALOGSERVER/supervisor.conf: Permission denied.
The permissions for this file are:
-rw------- 1 root root 2970 Jun 27 23:01 supervisor.conf
All other files and directories in the /var/run/cloudera-scm-agent/process/148-impala-CATLOGSERVER folder having owner impala and group impala.
When I change the permissions it will be overwritten on the next start.
What is wrong here?
And what needs to be changed. I recently installed a different service and changed the visudo file, but I have not changed the Impala configuration.
Thanks for the advice.
06-28-2017 10:05 PM
Would you mind deleting the Impala Catalog and Statestore instance . Add a new instance one more . see if those errors popping up.
i hardly remeber I had this issue . the above fixed it for me .
06-29-2017 01:02 AM
I have the same problem. I installed CDH 5.11.1 on a single machine AWS EC2. When starting impala service, Impala Catalog Server and Impala Daemon don't start. I removed the role and added again but the problem still persists.
Impala Daemon stderr :
+ exec /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/154-impala-IMPALAD/impala-conf/impalad_flags
Redirecting stderr to /var/log/impalad/impalad.ERROR
Can't open /run/cloudera-scm-agent/process/154-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/154-impala-IMPALAD/proc.json: Permission denied
Impala Catalog Server stderr:
exec /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../../bin/catalogd --flagfile=/run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/impala-conf/catalogserver_flags
Redirecting stderr to /var/log/catalogd/catalogd.ERROR
Can't open /run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/155-impala-CATALOGSERVER/proc.json: Permission denied
Any help will be appreciated.
06-29-2017 05:33 AM
since i see you logs says that you are downlading the parcels from /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/impala/../..
it says ( in your log file ) go to
it may direct to mini-dump logs (/var/log/impala-minidumps/impalad / )
if you open the file and if you come across the JRE runtime error plus if it complains about the core dump .
Then it is the bug that is related to Linux kernel , impala's JRE error .
the fix is to increase the Stack thread . do the below steps
the error should look like this
A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007fa6b9f80c18, pid=3819, tid=0x00007fa6cfdb4900 # # JRE version: (8.0_131-b11) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp #
The fix is
In Cloduera manager UI -> Cluster -> Go to Impala - Configuration tab .
Step 1 :
Impala Daemon properties:
Impala Daemon Environment Advanced Configuration Snippet (Safety Valve)
apply the JAVA_TOOL_OPTIONS=-Xss2m
Steo 2 :
Apply the same for Impala Catalog -
"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)"
before you restart the catalog daemon.
Reason - The crash was caused by a change to the linux kernel that modified the memory layout around JAVAthread stacks.
Please let me know if that works