cloudera manager will not start - fresh install

[#1]

Hello,

I have completed installation of Cloudera Enterprise 5.6.x on Red Hat Enterprise Linux Server release 6.7 in steps described as "installation path b" without errors.

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html#cmig_topic_...

Start of cloudera manager immediately fails - cloudera-scm-server.out only contains a few lines..

cd /var/log/cloudera-scm-server/
[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat *.out

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

Java JDK was installed as described here:

http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_jdk_installation.html?scroll=t...

Java environment for "hduser"..

echo $JAVA_HOME
/usr/java/jdk1.8.0_72

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ java -version
java version "1.8.0_72"

Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

I would appreciate any suggestions on how to proceed.

Thank you

[#2]

It appears you may not be starting Cloudera Manager when root or sudo. This is required and is likely the reason you are not allowed permission to java.

Try "service cloudera-scm-server start" as root or with sudo.

[#3]

Sorry I did not paste startup command - I'm definitely using sudo..

[hduser@slrs01va6f1clm1 ~]$ sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]

[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat cloudera-scm-server.out

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

[#4]

can provide the output for following command $ stat /usr/java/jdk1.8.0_72/bin/java 

[#5]

[hduser@slrs01va6f1clm1 ~]$ stat /usr/java/jdk1.8.0_72/bin/java

File: `/usr/java/jdk1.8.0_72/bin/java'
Size: 7734 Blocks: 16 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 419958 Links: 1
Access: (0776/-rwxrwxrw-) Uid: ( 504/ hduser) Gid: ( 504/ hadoop)
Access: 2016-03-24 18:08:14.042587642 -0500
Modify: 2015-12-23 00:10:09.000000000 -0600
Change: 2016-03-23 16:30:45.733564454 -0500

[#6]

strace output..

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ strace sudo service cloudera-scm-server start
execve("/usr/bin/sudo", ["sudo", "service", "cloudera-scm-server", "start"], [/* 25 vars */]) = 0
brk(0) = 0x7fb6a44d9000
fcntl(0, F_GETFD) = 0
fcntl(1, F_GETFD) = 0
fcntl(2, F_GETFD) = 0
access(0x7fb6a29af58c, F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb2000
access(0x7fb6a29b1280, R_OK) = -1 ENOENT (No such file or directory)
open(0x7fb6a29af901, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 34014, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb6a2ba9000
close(3) = 0
open(0x7fb6a2bb2640, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea78, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2236976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a2771000
mprotect(0x7fb6a2788000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a2988000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fb6a2988000
close(3) = 0
open(0x7fb6a2bb2b08, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea48, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2221912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a2552000
mprotect(0x7fb6a256f000, 2093056, PROT_NONE) = 0
mmap(0x7fb6a276e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7fb6a276e000
mmap(0x7fb6a2770000, 1880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2770000
close(3) = 0
open(0x7fb6a2bb2fd0, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea18, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba8000
mmap(NULL, 2105600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a234f000
mprotect(0x7fb6a2351000, 2093056, PROT_NONE) = 0
mmap(0x7fb6a2550000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fb6a2550000
close(3) = 0
open(0x7fb6a2ba84b0, O_RDONLY) = 3
read(3, 0x7ffeb5d5e9e8, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a214b000
mprotect(0x7fb6a214d000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a234d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb6a234d000
close(3) = 0
open(0x7fb6a2ba8978, O_RDONLY) = 3
read(3, 0x7ffeb5d5e9b8, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a1db7000
mprotect(0x7fb6a1f41000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a2141000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x7fb6a2141000
mmap(0x7fb6a2146000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2146000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba5000
arch_prctl(ARCH_SET_FS, 0x7fb6a2ba57a0) = 0
mprotect(0x7fb6a2141000, 16384, PROT_READ) = 0
mprotect(0x7fb6a234d000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2550000, 4096, PROT_READ) = 0
mprotect(0x7fb6a276e000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2988000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2dd2000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2bb3000, 4096, PROT_READ) = 0
munmap(0x7fb6a2ba9000, 34014) = 0
statfs(0x7fb6a25683d3, {...}) = 0
brk(0) = 0x7fb6a44d9000
brk(0x7fb6a44fa000) = 0x7fb6a44fa000
open(0x7fb6a25683b7, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb1000
read(3, 0x7fb6a2bb1000, 1024) = 304
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7fb6a2bb1000, 4096) = 0
open(0x7fb6a1f108e0, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb69bf26000
close(3) = 0
geteuid() = 504
open(0x7ffeb5d5eb20, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb1000
read(3, 0x7fb6a2bb1000, 4096) = 2512
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7fb6a2bb1000, 4096) = 0
open(0x7fb6a44da110, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db1a0, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44da1a0, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db120, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db230, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44da220, O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, 0x7ffeb5d5f82a, 4sudo) = 4
write(2, 0x7fb6a2bcc977, 2: ) = 2
write(2, 0x7ffeb5d5c650, 54effective uid is not 0, is sudo installed setuid root?) = 54
write(2, 0x7fb6a2145903, 1
) = 1
close(0) = 0
access(0x7fb6a1f0edc1, F_OK) = -1 ENOENT (No such file or directory)
open(0x7fb6a1f0edb3, O_RDONLY|O_CLOEXEC) = 0
fcntl(0, F_GETFD) = 0x1 (flags FD_CLOEXEC)
lseek(0, 0, SEEK_SET) = 0
alarm(0) = 0
rt_sigaction(SIGALRM, {...}, {...}, 😎 = 0
alarm(10) = 0
fcntl(0, F_SETLKW, {...}) = 0
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, "", 384) = 0
fcntl(0, F_SETLKW, {...}) = 0
alarm(0) = 10
rt_sigaction(SIGALRM, {...}, NULL, 😎 = 0
exit_group(1) = ?
+++ exited with 1 +++

[#7]

> write(2, 0x7ffeb5d5c650, 54effective uid is not 0, is sudo installed setuid root?) = 54

The above could indicate an issue also, looking into the "stat" output have you change the permission on /usr or only in /usr/java/...; can you check if you sudo works try this command sudo su - ; can you list your mount flags (cat /proc/mounts) ?

[#8]

sudo su -
[root@slrs01va6f1clm1 ~]# exit

[hduser@slrs01va6f1clm1 ~]$ cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=66060492k,nr_inodes=16515123,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/vglocal00-root00 / ext4 rw,relatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-home00 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-opt00 /opt ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-tmp00 /tmp ext4 rw,nosuid,nodev,noexec,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-log00 /var/log ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/3600a0980383036566e3f482f48633158 /data ext4 rw,noatime,barrier=1,stripe=16,data=ordered 0 0
cm_processes /var/run/cloudera-scm-agent/process tmpfs rw,relatime,mode=751 0 0
[hduser@slrs01va6f1clm1 ~]$

[#9]

I would like to uninstall the master node and start over.

I see multiple uninstall pages/procedures.

Can anyone recommend a procedure for uninstalling cloudera manager, agents and all packages for CDH 5.6.x?

Thank you

[#10]

I completely removed all packages and re-installed master node.

No change in behviour. Server start fails immediately..

[hduser@slrs01va6f1clm1 ~]$ sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]

[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat cloudera-scm-server.out

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

This is pretty frustrating.

I would appreciate any suggstions.

Thank you