Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Re: Accumulo Logger and Tablet server are not starting on some nodes

Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Explorer
telecom@telecom4:~$ sudo  ls -lah /run/cloudera-scm-agent/
total 0
drwxr-x--x  5 root root 120 Mar 26 11:16 .
drwxr-xr-x 27 root root 960 Mar 26 13:27 ..
drwxr-x--x  6 root root 120 Mar 26 11:16 cgroups
prw-------  1 root root   0 Mar 26 12:38 events
drwxr-x--x 22 root root 440 Mar 26 12:38 process
drwxr-x--x  3 root root 120 Mar 26 11:16 supervisor

 

7 REPLIES 7

Accumulo Logger and Tablet server are not starting on some nodes

Explorer

Hello,

 

I am trying to run 3 node cluster with cloudera manager. Currently hdfs and mapReduce services are running on all nodes properly. I am having issue in running accumulo logger and tablet server on some nodes and I am getting permission issue in the log file. I am not able to understand it. If somebody help me understand it, I will be gratefull.

 

Logger log File:

 

++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/bin/hadoop
++ export HIVE_DEFAULT_XML=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hive/conf/hive-default.xml
++ HIVE_DEFAULT_XML=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hive/conf/hive-default.xml
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/meta/accumulo_env.sh
+ PARCEL_DIRNAME=ACCUMULO-1.4.4-1.cdh4.5.0.p0.65
+ . /opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/meta/accumulo_env.sh
++ ACCUMULO_DIRNAME=ACCUMULO-1.4.4-1.cdh4.5.0.p0.65
++ export CDH_ACCUMULO_HOME=/opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/lib/accumulo
++ CDH_ACCUMULO_HOME=/opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/lib/accumulo
+ echo 'Using /run/cloudera-scm-agent/process/139-accumulo-ACCUMULO_LOGGER as conf dir'
+ echo 'Using scripts/accumulo.sh as process script'
+ chmod u+x /run/cloudera-scm-agent/process/139-accumulo-ACCUMULO_LOGGER/scripts/accumulo.sh
+ exec /run/cloudera-scm-agent/process/139-accumulo-ACCUMULO_LOGGER/scripts/accumulo.sh logger
/usr/lib/cmf/service/csd/csd.sh: line 23: /run/cloudera-scm-agent/process/139-accumulo-ACCUMULO_LOGGER/scripts/accumulo.sh: Permission denied
/usr/lib/cmf/service/csd/csd.sh: line 23: exec: /run/cloudera-scm-agent/process/139-accumulo-ACCUMULO_LOGGER/scripts/accumulo.sh: cannot execute: Permission denied

Tablet server log file:

 

++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop/bin/hadoop
++ export HIVE_DEFAULT_XML=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hive/conf/hive-default.xml
++ HIVE_DEFAULT_XML=/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hive/conf/hive-default.xml
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/meta/accumulo_env.sh
+ PARCEL_DIRNAME=ACCUMULO-1.4.4-1.cdh4.5.0.p0.65
+ . /opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/meta/accumulo_env.sh
++ ACCUMULO_DIRNAME=ACCUMULO-1.4.4-1.cdh4.5.0.p0.65
++ export CDH_ACCUMULO_HOME=/opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/lib/accumulo
++ CDH_ACCUMULO_HOME=/opt/cloudera/parcels/ACCUMULO-1.4.4-1.cdh4.5.0.p0.65/lib/accumulo
+ echo 'Using /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER as conf dir'
+ echo 'Using scripts/accumulo.sh as process script'
+ chmod u+x /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/accumulo.sh
+ exec /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/accumulo.sh tserver
/usr/lib/cmf/service/csd/csd.sh: line 23: /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/accumulo.sh: Permission denied
/usr/lib/cmf/service/csd/csd.sh: line 23: exec: /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/accumulo.sh: cannot execute: Permission denied
Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Expert Contributor

Hi Adnan!

 

What version of CM are you using?

 

Did CM handle the creation of users for Accumulo on oyur test cluster, or did you manually create them through some other process?

 

The proximal problem is that when CM's extensible code attempts to invoke the process script it is getting a permission problem despite having just attempted to set the executable bit on the same script. This is seen in your logs as a "permission denied" error from the script csd.sh.

Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Explorer

I am using CM 5 beta 2. I didn't create users for Accumulo. I installed Accumulo using parcels method.

I checked for permission issues, all these scripts owner was Accumulo and they all were executable.

What should I do to solve this issue?

Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Expert Contributor

Can you go to that same failed node and copy/paste the ouput of the following commands into a "insert code" box like the one below?

 

$> ls -lah /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/

$> ls -lah /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/

$> ulimit -a

 

Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Explorer

Following is the output of above commands:

telecom@telecom4:~$ sudo  ls -lah /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/
total 12K
drwxr-x--x  5 accumulo accumulo 160 Mar 26 11:40 .
drwxr-x--x 22 root     root     440 Mar 26 12:38 ..
-rw-r-----  1 accumulo accumulo 925 Mar 26 11:40 accumulo-site.xml
-rw-r-----  1 accumulo accumulo 201 Mar 26 11:40 cloudera-monitor.properties
drwxr-x--x  2 accumulo accumulo 140 Mar 26 11:40 hadoop-conf
-rw-r-----  1 accumulo accumulo 712 Mar 26 11:40 log4j.properties
drwxr-x--x  2 accumulo accumulo  80 Mar 26 11:40 logs
drwxr-x--x  2 accumulo accumulo 120 Mar 26 11:40 scripts
telecom@telecom4:~$ ls -lah /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/
ls: cannot open directory /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/: Permission denied
telecom@telecom4:~$ sudo ls -lah /run/cloudera-scm-agent/process/140-accumulo-ACCUMULO_TSERVER/scripts/
total 16K
drwxr-x--x 2 accumulo accumulo  120 Mar 26 11:40 .
drwxr-x--x 5 accumulo accumulo  160 Mar 26 11:40 ..
-rw-r----- 1 accumulo accumulo  169 Mar 26 11:40 accumulo-env.sh
-rw-r----- 1 accumulo accumulo 2.2K Mar 26 11:40 accumulo-metrics.xml
-rwxr----- 1 accumulo accumulo 3.0K Mar 26 11:40 accumulo.sh
-rw-r----- 1 accumulo accumulo 1.6K Mar 26 11:40 log4j.properties
telecom@telecom4:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 60881
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 60881
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

 

Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Expert Contributor

One more real quick:

 

$ sudo  ls -lah /run/cloudera-scm-agent/

 

Highlighted

Re: Accumulo Logger and Tablet server are not starting on some nodes

Expert Contributor

FYI I'm trying to replicate on a 3 node cluster I have locally for testing. So far everything just works, but this was an upgrade from a non-CM instance.

 

I'll let you know once I have a follow on question.

Don't have an account?
Coming from Hortonworks? Activate your account here