question Re: cloudera manager will not start - fresh install in Support Questions

cloudera manager will not start - fresh install

Romulus — Fri, 16 Sep 2022 10:10:51 GMT

Hello,

I have completed installation of Cloudera Enterprise 5.6.x on Red Hat Enterprise Linux Server release 6.7 in steps described as "installation path b" without errors.

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html#cmig_topic_6_6_1_unique_1__section_mzw_l1d_d5_unique_3

Start of cloudera manager immediately fails - cloudera-scm-server.out only contains a few lines..

cd /var/log/cloudera-scm-server/
[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat *.out

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

Java JDK was installed as described here:

http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_jdk_installation.html?scroll=topic_29

Java environment for "hduser"..

echo $JAVA_HOME
/usr/java/jdk1.8.0_72

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ java -version
java version "1.8.0_72"

Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

I would appreciate any suggestions on how to proceed.

Thank you

Re: cloudera manager will not start - fresh install

bgooley — Thu, 24 Mar 2016 22:55:08 GMT

It appears you may not be starting Cloudera Manager when root or sudo. This is required and is likely the reason you are not allowed permission to java.

Try "service cloudera-scm-server start" as root or with sudo.

Re: cloudera manager will not start - fresh install

Romulus — Thu, 24 Mar 2016 23:50:30 GMT

Sorry I did not paste startup command - I'm definitely using sudo..

[hduser@slrs01va6f1clm1 ~]$ sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]

[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat cloudera-scm-server.out

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

Re: cloudera manager will not start - fresh install

michalis — Fri, 25 Mar 2016 00:43:48 GMT

can provide the output for following command $ stat /usr/java/jdk1.8.0_72/bin/java

Re: cloudera manager will not start - fresh install

Romulus — Fri, 25 Mar 2016 15:22:21 GMT

[hduser@slrs01va6f1clm1 ~]$ stat /usr/java/jdk1.8.0_72/bin/java

File: `/usr/java/jdk1.8.0_72/bin/java'
Size: 7734 Blocks: 16 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 419958 Links: 1
Access: (0776/-rwxrwxrw-) Uid: ( 504/ hduser) Gid: ( 504/ hadoop)
Access: 2016-03-24 18:08:14.042587642 -0500
Modify: 2015-12-23 00:10:09.000000000 -0600
Change: 2016-03-23 16:30:45.733564454 -0500

Re: cloudera manager will not start - fresh install

Romulus — Fri, 25 Mar 2016 17:11:41 GMT

strace output..

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ strace sudo service cloudera-scm-server start
execve("/usr/bin/sudo", ["sudo", "service", "cloudera-scm-server", "start"], [/* 25 vars */]) = 0
brk(0) = 0x7fb6a44d9000
fcntl(0, F_GETFD) = 0
fcntl(1, F_GETFD) = 0
fcntl(2, F_GETFD) = 0
access(0x7fb6a29af58c, F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb2000
access(0x7fb6a29b1280, R_OK) = -1 ENOENT (No such file or directory)
open(0x7fb6a29af901, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 34014, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb6a2ba9000
close(3) = 0
open(0x7fb6a2bb2640, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea78, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2236976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a2771000
mprotect(0x7fb6a2788000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a2988000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fb6a2988000
close(3) = 0
open(0x7fb6a2bb2b08, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea48, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2221912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a2552000
mprotect(0x7fb6a256f000, 2093056, PROT_NONE) = 0
mmap(0x7fb6a276e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7fb6a276e000
mmap(0x7fb6a2770000, 1880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2770000
close(3) = 0
open(0x7fb6a2bb2fd0, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea18, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba8000
mmap(NULL, 2105600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a234f000
mprotect(0x7fb6a2351000, 2093056, PROT_NONE) = 0
mmap(0x7fb6a2550000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fb6a2550000
close(3) = 0
open(0x7fb6a2ba84b0, O_RDONLY) = 3
read(3, 0x7ffeb5d5e9e8, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a214b000
mprotect(0x7fb6a214d000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a234d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb6a234d000
close(3) = 0
open(0x7fb6a2ba8978, O_RDONLY) = 3
read(3, 0x7ffeb5d5e9b8, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a1db7000
mprotect(0x7fb6a1f41000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a2141000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x7fb6a2141000
mmap(0x7fb6a2146000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2146000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba5000
arch_prctl(ARCH_SET_FS, 0x7fb6a2ba57a0) = 0
mprotect(0x7fb6a2141000, 16384, PROT_READ) = 0
mprotect(0x7fb6a234d000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2550000, 4096, PROT_READ) = 0
mprotect(0x7fb6a276e000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2988000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2dd2000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2bb3000, 4096, PROT_READ) = 0
munmap(0x7fb6a2ba9000, 34014) = 0
statfs(0x7fb6a25683d3, {...}) = 0
brk(0) = 0x7fb6a44d9000
brk(0x7fb6a44fa000) = 0x7fb6a44fa000
open(0x7fb6a25683b7, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb1000
read(3, 0x7fb6a2bb1000, 1024) = 304
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7fb6a2bb1000, 4096) = 0
open(0x7fb6a1f108e0, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb69bf26000
close(3) = 0
geteuid() = 504
open(0x7ffeb5d5eb20, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb1000
read(3, 0x7fb6a2bb1000, 4096) = 2512
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7fb6a2bb1000, 4096) = 0
open(0x7fb6a44da110, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db1a0, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44da1a0, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db120, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db230, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44da220, O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, 0x7ffeb5d5f82a, 4sudo) = 4
write(2, 0x7fb6a2bcc977, 2: ) = 2
write(2, 0x7ffeb5d5c650, 54effective uid is not 0, is sudo installed setuid root?) = 54
write(2, 0x7fb6a2145903, 1
) = 1
close(0) = 0
access(0x7fb6a1f0edc1, F_OK) = -1 ENOENT (No such file or directory)
open(0x7fb6a1f0edb3, O_RDONLY|O_CLOEXEC) = 0
fcntl(0, F_GETFD) = 0x1 (flags FD_CLOEXEC)
lseek(0, 0, SEEK_SET) = 0
alarm(0) = 0
rt_sigaction(SIGALRM, {...}, {...}, 😎 = 0
alarm(10) = 0
fcntl(0, F_SETLKW, {...}) = 0
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, "", 384) = 0
fcntl(0, F_SETLKW, {...}) = 0
alarm(0) = 10
rt_sigaction(SIGALRM, {...}, NULL, 😎 = 0
exit_group(1) = ?
+++ exited with 1 +++

Re: cloudera manager will not start - fresh install

michalis — Fri, 25 Mar 2016 20:15:10 GMT

> write(2, 0x7ffeb5d5c650, 54effective uid is not 0, is sudo installed setuid root?) = 54

The above could indicate an issue also, looking into the "stat" output have you change the permission on /usr or only in /usr/java/...; can you check if you sudo works try this command sudo su - ; can you list your mount flags (cat /proc/mounts) ?

Re: cloudera manager will not start - fresh install

Romulus — Fri, 25 Mar 2016 20:12:32 GMT

sudo su -
[root@slrs01va6f1clm1 ~]# exit

[hduser@slrs01va6f1clm1 ~]$ cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=66060492k,nr_inodes=16515123,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/vglocal00-root00 / ext4 rw,relatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-home00 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-opt00 /opt ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-tmp00 /tmp ext4 rw,nosuid,nodev,noexec,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-log00 /var/log ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/3600a0980383036566e3f482f48633158 /data ext4 rw,noatime,barrier=1,stripe=16,data=ordered 0 0
cm_processes /var/run/cloudera-scm-agent/process tmpfs rw,relatime,mode=751 0 0
[hduser@slrs01va6f1clm1 ~]$

Re: cloudera manager will not start - fresh install

Romulus — Fri, 25 Mar 2016 21:59:12 GMT

I would like to uninstall the master node and start over.

I see multiple uninstall pages/procedures.

Can anyone recommend a procedure for uninstalling cloudera manager, agents and all packages for CDH 5.6.x?

Thank you

Re: cloudera manager will not start - fresh install

Romulus — Mon, 28 Mar 2016 20:00:37 GMT

I completely removed all packages and re-installed master node.

No change in behviour. Server start fails immediately..

[hduser@slrs01va6f1clm1 ~]$ sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]

[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat cloudera-scm-server.out

This is pretty frustrating.

I would appreciate any suggstions.

Thank you

Re: cloudera manager will not start - fresh install

bgooley — Mon, 28 Mar 2016 21:01:34 GMT

Looking back at your permissions, it appears that you have the following as reported by "stat /usr/java/jdk1.8.0_72/bin/java":

Access: (0776/-rwxrwxrw-) Uid: ( 504/ hduser) Gid: ( 504/ hadoop)

Since the shell used to start Clouder Manager is owned by cloudera-scm, this would mean that unless you altered your init script to start as user "hduser" then "other" users don't have access to run java.

Try adding the execute permission for other on your java binary and try starting Cloudera Manager.

Regards,

Ben

Re: cloudera manager will not start - fresh install

Romulus — Tue, 29 Mar 2016 07:07:28 GMT

Thanks for the quick response. You're suggestion allowed to inch closer, but new problem exists..

[hduser@slrs01va6f1clm1 ~]$ sudo sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]

[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/
[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat *.out

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/java/jdk1.8.0_72/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[hduser@slrs01va6f1clm1 cloudera-scm-server]$

I created symlink for java and ensured shared library is recognized for access by root user..

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cd /usr/bin
[hduser@slrs01va6f1clm1 bin]$ ls -l java

lrwxrwxrwx 1 root root 30 Mar 29 01:26 java -> /usr/java/jdk1.8.0_72/bin/java

[hduser@slrs01va6f1clm1 bin]$ sudo java -version
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

[hduser@slrs01va6f1clm1 bin]$ sudo ldd /usr/java/jdk1.8.0_72/bin/java
linux-vdso.so.1 => (0x00007ffca5db1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038b0800000)
libjli.so => /usr/java/jdk1.8.0_72/bin/../lib/amd64/jli/libjli.so (0x00007f5ac2111000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003d6ec00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003d6e400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d6e000000)
[hduser@slrs01va6f1clm1 bin]$

[hduser@slrs01va6f1clm1 bin]$ sudo cat /etc/ld.so.conf.d/java.conf
/usr/java/jdk1.8.0_72/lib/amd64/jli

[hduser@slrs01va6f1clm1 bin]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hduser/bin:/usr/java/jdk1.8.0_72/bin:/usr/java/jdk1.8.0_72/lib/amd64/jli

[hduser@slrs01va6f1clm1 bin]$ echo $LD_LIBRARY_PATH
/usr/java/jdk1.8.0_72/lib/amd64/jli

Any suggestions?

Much appreciated.

Thanks!

Re: cloudera manager will not start - fresh install

bgooley — Wed, 30 Mar 2016 04:30:47 GMT

It appears that cloudera-scm still cannot access files in the java directories. Check that the file is read/execute by other.

This is mine:

-rwxr-xr-x 1 root root 106366 Sep 2 2014 libjli.so

I recommend just doing the following as again it appears there are permission problems in your JDK

chmod -R 777 /usr/java/jdk1.8.0_72

I'm not sure why permissions are like they are, but normally java is accessible by "other" I believe.

Regards,

Ben

Re: cloudera manager will not start - fresh install

Romulus — Fri, 22 Apr 2016 22:12:34 GMT

Suggested chmod applied, but problem persisted.

Solution was to re-install with 5.5.2.

Thanks for your replies.

Re: cloudera manager will not start - fresh install

DataSplainer — Mon, 25 Apr 2016 16:23:22 GMT

It looks like you installed Java as hduser instead of as root. This will prevent other users from accessing it.

Could you try uninstalling java, and re-installing it as root?

Re: cloudera manager will not start - fresh install

safal — Sat, 05 May 2018 20:59:33 GMT

I got the same problem when I put the Java files in my home directory. It only worked after I put the Java files in the /usr/java directory.