Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

cloudera manager will not start - fresh install

Explorer

Hello,

 

I have completed installation of Cloudera Enterprise 5.6.x on Red Hat Enterprise Linux Server release 6.7 in steps described as "installation path b" without errors.

 

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html#cmig_topic_...

 

Start of cloudera manager immediately fails - cloudera-scm-server.out only contains a few lines..

 

cd /var/log/cloudera-scm-server/
[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat *.out


JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

 

Java JDK was installed as described here:

 

http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_jdk_installation.html?scroll=t...

 

Java environment for "hduser"..

 

echo $JAVA_HOME
/usr/java/jdk1.8.0_72


[hduser@slrs01va6f1clm1 cloudera-scm-server]$ java -version
java version "1.8.0_72"


Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

 

I would appreciate any suggestions on how to proceed.

 

Thank you

1 ACCEPTED SOLUTION

Explorer

Suggested chmod applied, but problem persisted.

 

Solution was to re-install with 5.5.2.

 

Thanks for your replies.

View solution in original post

15 REPLIES 15

Super Guru

It appears you may not be starting Cloudera Manager when root or sudo. This is required and is likely the reason you are not allowed permission to java.

 

Try "service cloudera-scm-server start" as root or with sudo.

Explorer

 

Sorry I did not paste startup command - I'm definitely using sudo..


[hduser@slrs01va6f1clm1 ~]$ sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]


[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/

 

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat cloudera-scm-server.out


JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

Super Collaborator

can provide the output for following command $ stat /usr/java/jdk1.8.0_72/bin/java 

Explorer

 

[hduser@slrs01va6f1clm1 ~]$ stat /usr/java/jdk1.8.0_72/bin/java


File: `/usr/java/jdk1.8.0_72/bin/java'
Size: 7734 Blocks: 16 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 419958 Links: 1
Access: (0776/-rwxrwxrw-) Uid: ( 504/ hduser) Gid: ( 504/ hadoop)
Access: 2016-03-24 18:08:14.042587642 -0500
Modify: 2015-12-23 00:10:09.000000000 -0600
Change: 2016-03-23 16:30:45.733564454 -0500

Explorer

strace output..

 

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ strace sudo service cloudera-scm-server start
execve("/usr/bin/sudo", ["sudo", "service", "cloudera-scm-server", "start"], [/* 25 vars */]) = 0
brk(0) = 0x7fb6a44d9000
fcntl(0, F_GETFD) = 0
fcntl(1, F_GETFD) = 0
fcntl(2, F_GETFD) = 0
access(0x7fb6a29af58c, F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb2000
access(0x7fb6a29b1280, R_OK) = -1 ENOENT (No such file or directory)
open(0x7fb6a29af901, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 34014, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb6a2ba9000
close(3) = 0
open(0x7fb6a2bb2640, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea78, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2236976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a2771000
mprotect(0x7fb6a2788000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a2988000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fb6a2988000
close(3) = 0
open(0x7fb6a2bb2b08, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea48, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2221912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a2552000
mprotect(0x7fb6a256f000, 2093056, PROT_NONE) = 0
mmap(0x7fb6a276e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7fb6a276e000
mmap(0x7fb6a2770000, 1880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2770000
close(3) = 0
open(0x7fb6a2bb2fd0, O_RDONLY) = 3
read(3, 0x7ffeb5d5ea18, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba8000
mmap(NULL, 2105600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a234f000
mprotect(0x7fb6a2351000, 2093056, PROT_NONE) = 0
mmap(0x7fb6a2550000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fb6a2550000
close(3) = 0
open(0x7fb6a2ba84b0, O_RDONLY) = 3
read(3, 0x7ffeb5d5e9e8, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a214b000
mprotect(0x7fb6a214d000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a234d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb6a234d000
close(3) = 0
open(0x7fb6a2ba8978, O_RDONLY) = 3
read(3, 0x7ffeb5d5e9b8, 832) = 832
fstat(3, {...}) = 0
mmap(NULL, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb6a1db7000
mprotect(0x7fb6a1f41000, 2097152, PROT_NONE) = 0
mmap(0x7fb6a2141000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x7fb6a2141000
mmap(0x7fb6a2146000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2146000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2ba5000
arch_prctl(ARCH_SET_FS, 0x7fb6a2ba57a0) = 0
mprotect(0x7fb6a2141000, 16384, PROT_READ) = 0
mprotect(0x7fb6a234d000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2550000, 4096, PROT_READ) = 0
mprotect(0x7fb6a276e000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2988000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2dd2000, 4096, PROT_READ) = 0
mprotect(0x7fb6a2bb3000, 4096, PROT_READ) = 0
munmap(0x7fb6a2ba9000, 34014) = 0
statfs(0x7fb6a25683d3, {...}) = 0
brk(0) = 0x7fb6a44d9000
brk(0x7fb6a44fa000) = 0x7fb6a44fa000
open(0x7fb6a25683b7, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb1000
read(3, 0x7fb6a2bb1000, 1024) = 304
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7fb6a2bb1000, 4096) = 0
open(0x7fb6a1f108e0, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb69bf26000
close(3) = 0
geteuid() = 504
open(0x7ffeb5d5eb20, O_RDONLY) = 3
fstat(3, {...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb6a2bb1000
read(3, 0x7fb6a2bb1000, 4096) = 2512
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7fb6a2bb1000, 4096) = 0
open(0x7fb6a44da110, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db1a0, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44da1a0, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db120, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44db230, O_RDONLY) = -1 ENOENT (No such file or directory)
open(0x7fb6a44da220, O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, 0x7ffeb5d5f82a, 4sudo) = 4
write(2, 0x7fb6a2bcc977, 2: ) = 2
write(2, 0x7ffeb5d5c650, 54effective uid is not 0, is sudo installed setuid root?) = 54
write(2, 0x7fb6a2145903, 1
) = 1
close(0) = 0
access(0x7fb6a1f0edc1, F_OK) = -1 ENOENT (No such file or directory)
open(0x7fb6a1f0edb3, O_RDONLY|O_CLOEXEC) = 0
fcntl(0, F_GETFD) = 0x1 (flags FD_CLOEXEC)
lseek(0, 0, SEEK_SET) = 0
alarm(0) = 0
rt_sigaction(SIGALRM, {...}, {...}, 😎 = 0
alarm(10) = 0
fcntl(0, F_SETLKW, {...}) = 0
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, 0x7fb6a214a080, 384) = 384
read(0, "", 384) = 0
fcntl(0, F_SETLKW, {...}) = 0
alarm(0) = 10
rt_sigaction(SIGALRM, {...}, NULL, 😎 = 0
exit_group(1) = ?
+++ exited with 1 +++

Super Collaborator

write(2, 0x7ffeb5d5c650, 54effective uid is not 0, is sudo installed setuid root?) = 54

The above could indicate an issue also, looking into the "stat" output have you change the permission on /usr or only in /usr/java/...; can you check if you sudo works try this command sudo su - ; can you list your mount flags (cat /proc/mounts) ?

Explorer

 

sudo su -
[root@slrs01va6f1clm1 ~]# exit

 

[hduser@slrs01va6f1clm1 ~]$ cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=66060492k,nr_inodes=16515123,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/vglocal00-root00 / ext4 rw,relatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-home00 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-opt00 /opt ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-tmp00 /tmp ext4 rw,nosuid,nodev,noexec,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vglocal00-log00 /var/log ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/3600a0980383036566e3f482f48633158 /data ext4 rw,noatime,barrier=1,stripe=16,data=ordered 0 0
cm_processes /var/run/cloudera-scm-agent/process tmpfs rw,relatime,mode=751 0 0
[hduser@slrs01va6f1clm1 ~]$

Explorer

I would like to uninstall the master node and start over.

 

I see multiple uninstall pages/procedures.

 

Can anyone recommend a procedure for uninstalling cloudera manager, agents and all packages for CDH 5.6.x?

 

Thank you

Explorer

I completely removed all packages and re-installed master node.

 

No change in behviour. Server start fails immediately..

 

[hduser@slrs01va6f1clm1 ~]$ sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]


[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/

 

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat cloudera-scm-server.out


JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/share/cmf/bin/cmf-server: line 129: /usr/java/jdk1.8.0_72/bin/java: Permission denied
/usr/share/cmf/bin/cmf-server: line 129: exec: /usr/java/jdk1.8.0_72/bin/java: cannot execute: Permission denied

 

This is pretty frustrating.

 

I would appreciate any suggstions.

 

Thank you

Super Guru

 

Looking back at your permissions, it appears that you have the following as reported by "stat /usr/java/jdk1.8.0_72/bin/java":

 

Access: (0776/-rwxrwxrw-) Uid: ( 504/ hduser) Gid: ( 504/ hadoop)

 

Since the shell used to start Clouder Manager is owned by cloudera-scm, this would mean that unless you altered your init script to start as user "hduser" then "other" users don't have access to run java.

 

Try adding the execute permission for other on your java binary and try starting Cloudera Manager.

 

Regards,

 

Ben

Explorer

Thanks for the quick response. You're suggestion allowed to inch closer, but new problem exists..

 

[hduser@slrs01va6f1clm1 ~]$ sudo sudo service cloudera-scm-server start
[sudo] password for hduser:
Starting cloudera-scm-server: [FAILED]


[hduser@slrs01va6f1clm1 ~]$ cd /var/log/cloudera-scm-server/
[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cat *.out

 

JAVA_HOME=/usr/java/jdk1.8.0_72
/usr/java/jdk1.8.0_72/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[hduser@slrs01va6f1clm1 cloudera-scm-server]$

 

I created symlink for java and ensured shared library is recognized for access by root user..

 

[hduser@slrs01va6f1clm1 cloudera-scm-server]$ cd /usr/bin
[hduser@slrs01va6f1clm1 bin]$ ls -l java


lrwxrwxrwx 1 root root 30 Mar 29 01:26 java -> /usr/java/jdk1.8.0_72/bin/java

 

[hduser@slrs01va6f1clm1 bin]$ sudo java -version
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

 

 

[hduser@slrs01va6f1clm1 bin]$ sudo ldd /usr/java/jdk1.8.0_72/bin/java
linux-vdso.so.1 => (0x00007ffca5db1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038b0800000)
libjli.so => /usr/java/jdk1.8.0_72/bin/../lib/amd64/jli/libjli.so (0x00007f5ac2111000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003d6ec00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003d6e400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d6e000000)
[hduser@slrs01va6f1clm1 bin]$

 

[hduser@slrs01va6f1clm1 bin]$ sudo cat /etc/ld.so.conf.d/java.conf
/usr/java/jdk1.8.0_72/lib/amd64/jli

 

[hduser@slrs01va6f1clm1 bin]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hduser/bin:/usr/java/jdk1.8.0_72/bin:/usr/java/jdk1.8.0_72/lib/amd64/jli


[hduser@slrs01va6f1clm1 bin]$ echo $LD_LIBRARY_PATH
/usr/java/jdk1.8.0_72/lib/amd64/jli

Any suggestions?

 

Much appreciated.

 

Thanks!

 

Super Guru

It appears that cloudera-scm still cannot access files in the java directories.  Check that the file is read/execute by other.

This is mine:

 

-rwxr-xr-x 1 root root 106366 Sep 2 2014 libjli.so

 

I recommend just doing the following as again it appears there are permission problems in your JDK

 

chmod -R 777 /usr/java/jdk1.8.0_72

 

I'm not sure why permissions are like they are, but normally java is accessible by "other" I believe.

 

Regards,

 

Ben

Explorer

Suggested chmod applied, but problem persisted.

 

Solution was to re-install with 5.5.2.

 

Thanks for your replies.

It looks like you installed Java as hduser instead of as root. This will prevent other users from accessing it.

 

Could you try uninstalling java, and re-installing it as root?

New Contributor

I got the same problem when I put the Java files in my home directory. It only worked after I put the Java files in the /usr/java directory.