Created on 11-21-2016 08:21 AM - edited 11-21-2016 08:23 AM
Hi Guys,
My System:
RHEL 7.2, 48GB RAM, CM 5.8, CDH 5.7.4, Kerberos Enabled
I have a problem with a cloudera agent. I am not able to run it. It gets to some loop which looks as follow:
21/Nov/2016 15:20:17 +0000] 12974 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro [21/Nov/2016 15:20:17 +0000] 12974 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Agent does create users/groups and apply file permissions [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Loading parcel manifest for: CDH-5.7.4-1.cdh5.7.4.p0.2 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Loading parcel manifest for: KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Loading parcel manifest for: ARCADIAENTERPRISE-3.2.1.0-1477612993.cdh5 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Ensuring users/groups exist for new parcel CDH-5.7.4-1.cdh5.7.4.p0.2. [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hadoop'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'flume'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hdfs'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'solr'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'zookeeper'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'llama'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'httpfs'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'mapred'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'sqoop'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'yarn'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'kms'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hive'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'sqoop2'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'oozie'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hbase'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'sentry'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'impala'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'spark'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hue'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'flume', '-K', 'UMASK=022', '--home', '/var/lib/flume-ng', '--comment', 'Flume', '--shell', '/bin/false', 'flume'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'hdfs', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-hdfs', '--comment', 'Hadoop HDFS', '--shell', '/bin/bash', '--groups', 'hadoop', 'hdfs'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'solr', '-K', 'UMASK=022', '--home', '/var/lib/solr', '--comment', 'Solr', '--shell', '/sbin/nologin', 'solr'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'zookeeper', '-K', 'UMASK=022', '--home', '/var/lib/zookeeper', '--comment', 'ZooKeeper', '--shell', '/bin/false', 'zookeeper'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'llama', '-K', 'UMASK=022', '--home', '/var/lib/llama', '--comment', 'Llama', '--shell', '/bin/bash', '--groups', 'llama', 'llama'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'httpfs', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-httpfs', '--comment', 'Hadoop HTTPFS', '--shell', '/bin/bash', 'httpfs'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'mapred', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-mapreduce', '--comment', 'Hadoop MapReduce', '--shell', '/bin/bash', '--groups', 'hadoop', 'mapred'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'sqoop', '-K', 'UMASK=022', '--home', '/var/lib/sqoop', '--comment', 'Sqoop', '--shell', '/bin/false', 'sqoop'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'yarn', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-yarn', '--comment', 'Hadoop Yarn', '--shell', '/bin/bash', '--groups', 'hadoop', 'yarn'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'kms', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-kms', '--comment', 'Hadoop KMS', '--shell', '/bin/bash', 'kms'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'hive', '-K', 'UMASK=022', '--home', '/var/lib/hive', '--comment', 'Hive', '--shell', '/bin/false', 'hive'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'sqoop2', '-K', 'UMASK=022', '--home', '/var/lib/sqoop2', '--comment', 'Sqoop 2 User', '--shell', '/sbin/nologin', '--groups', 'sqoop', 'sqoop2'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'oozie', '-K', 'UMASK=022', '--home', '/var/lib/oozie', '--comment', 'Oozie User', '--shell', '/bin/false', 'oozie'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'hbase', '-K', 'UMASK=022', '--home', '/var/lib/hbase', '--comment', 'HBase', '--shell', '/bin/false', 'hbase'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'sentry', '-K', 'UMASK=022', '--home', '/var/lib/sentry', '--comment', 'Sentry', '--shell', '/sbin/nologin', 'sentry'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'impala', '-K', 'UMASK=022', '--home', '/var/lib/impala', '--comment', 'Impala', '--shell', '/bin/bash', '--groups', 'hive', 'impala'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'spark', '-K', 'UMASK=022', '--home', '/var/lib/spark', '--comment', 'Spark', '--shell', '/sbin/nologin', 'spark'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'hue', '-K', 'UMASK=022', '--home', '/usr/lib/hue', '--comment', 'Hue', '--shell', '/bin/false', 'hue'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Ensuring correct file permissions for new parcel CDH-5.7.4-1.cdh5.7.4.p0.2. [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-yarn/bin/container-executor root yarn [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'root:yarn', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-yarn/bin/container-executor'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-yarn/bin/container-executor 6050 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '6050', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-yarn/bin/container-executor'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.dist/conf/server.xml sqoop2 sqoop [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'sqoop2:sqoop', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.dist/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.dist/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.dist/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.dist/conf/server.xml solr solr [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'solr:solr', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.dist/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.dist/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.dist/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop/desktop.db hue hue [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop/desktop.db'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop/desktop.db 0660 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0660', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop/desktop.db'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller root mapred [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'root:mapred', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller 4754 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '4754', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml httpfs httpfs [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'httpfs:httpfs', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml oozie oozie [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.mr1/conf/server.xml sqoop2 sqoop [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'sqoop2:sqoop', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.mr1/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.mr1/conf/server.xml 0750 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/sqoop2/tomcat-conf.mr1/conf/server.xml'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop hue hue [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop 0755 [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0755', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/hue/desktop'] [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml httpfs httpfs [21/Nov/2016 15:20:17 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'httpfs:httpfs', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml 0750 [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/server.xml oozie oozie [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/server.xml 0750 [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https/conf/ssl/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/server.xml oozie oozie [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/server.xml 0750 [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.https.mr1/conf/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.https/conf/server.xml solr solr [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'solr:solr', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.https/conf/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.https/conf/server.xml 0750 [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/solr/tomcat-conf.https/conf/server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml oozie oozie [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml 0750 [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Ensuring users/groups exist for new parcel KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17. [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Ensuring correct file permissions for new parcel KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17. [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chown: /opt/cloudera/parcels/KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17/var/lib/kms-keytrustee kms kms [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chown', 'kms:kms', u'/opt/cloudera/parcels/KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17/var/lib/kms-keytrustee'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO chmod: /opt/cloudera/parcels/KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17/var/lib/kms-keytrustee 0755 [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['chmod', '0755', u'/opt/cloudera/parcels/KEYTRUSTEE-5.9.0-5.KEYTRUSTEE5.9.0.p0.17/var/lib/kms-keytrustee'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Ensuring users/groups exist for new parcel ARCADIAENTERPRISE-3.2.1.0-1477612993.cdh5. [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'arcadia'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'arcadia'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'arcadia', '-K', 'UMASK=022', '--home', '/var/lib/arcadia', '--comment', 'Arcadia Data', '--shell', '/bin/bash', '--groups', 'arcadia,hive,hdfs,impala', 'arcadia'] [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel INFO Ensuring correct file permissions for new parcel ARCADIAENTERPRISE-3.2.1.0-1477612993.cdh5. [21/Nov/2016 15:20:18 +0000] 12974 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache [21/Nov/2016 15:20:18 +0000] 12974 MainThread agent ERROR Caught unexpected exception in main loop. Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 710, in start self._init_after_first_heartbeat_response(resp_data) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 840, in _init_after_first_heartbeat_response self.client_configs.load() File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 682, in load new_deployed.update(self._lookup_alternatives(fname)) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 432, in _lookup_alternatives return self._parse_alternatives(alt_name, out) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 444, in _parse_alternatives path, _, _, priority_str = line.rstrip().split(" ") ValueError: too many values to unpack
I think the line with
ValueError: too many values to unpack
is a key here but no idea what to do with it right now.
Also, I tried to run agent in DEBUG mode from:
/etc/cloudera-scm-agent/config.ini
but changing it there doesn't affect the agent at all........
One more thing, that happen only on Gateway Node. The rest of the nodes are just fine.
Thanks,
Andrzej
Created 11-21-2016 10:57 AM
You might do something like this:
# alternatives --list | awk '{print $1}' | xargs -n1 alternatives --display |grep "^/" > all_alternatives.txt
# awk -F ' ' -v NCOLS=4 'NF!=NCOLS{printf "Wrong number of columns at line %d\n", NR}' all_alternatives.txt
The above will list out all the alternatives display information and write that to a file
Then, it will check to see if there are any lines with more or less than 4 columns and print out the line number.
You could then use that line number to reference the all_alternatives.txt file line.
That should show us which alternative is tripping up the agent.
Created 10-04-2017 06:53 AM
If anyone is interested i've updated to CM 5.12.1 and the issue has been fixed.
Created 11-21-2016 09:31 AM
Hello,
The exception occurs when the agent is attempting to run the "alternatives" command to list out all the current alternatives. The agent code parses the output and assigns it to variables. The exception indicates that one of the rows returned by "alternatives" has more columns than expected.
We see this assignment:
path, _, _, priority_str = line.rstrip().split(" ")
We have found that the OpenJDK name may have a space in the name so it has an extra column.
We don't have a fix now, but you could remove OpenJDK from alternatives if that is the one with the extra column.
I'll make sure we tighten up the code to prevent this sort of issue in the future.
Created on 11-21-2016 09:40 AM - edited 11-21-2016 09:40 AM
I don't have any OpenJDK packages installed.
Only Oracle jdk 1.8.
Created 11-21-2016 09:49 AM
Understood. That was just a guess on which alternative may be causing the problem. I'm not sure how to get alternatives to list the "--display" style output when just listing. We may need to do some scripting to iterate over each alternative and write out the --display details for each to see which has more than 4 columns.
Alternatively, the only option would be to make code changes.
Created 11-21-2016 10:57 AM
You might do something like this:
# alternatives --list | awk '{print $1}' | xargs -n1 alternatives --display |grep "^/" > all_alternatives.txt
# awk -F ' ' -v NCOLS=4 'NF!=NCOLS{printf "Wrong number of columns at line %d\n", NR}' all_alternatives.txt
The above will list out all the alternatives display information and write that to a file
Then, it will check to see if there are any lines with more or less than 4 columns and print out the line number.
You could then use that line number to reference the all_alternatives.txt file line.
That should show us which alternative is tripping up the agent.
Created on 11-22-2016 01:28 AM - edited 11-22-2016 01:31 AM
I'm getting same issue. And i have removed OpenJDK from alternatives. Then i restarted the CDH server and agent, no luck. It is freeze on Activated state. Any help
Created 04-26-2017 02:44 AM
Please eloborate the fix , Using centos 7 and Cloudera Manager 5.11 . I'm not sure what can I do to resolve the issue .
Created on 01-04-2017 11:54 AM - edited 01-04-2017 12:36 PM
i experienced this issue also. removing open jdk resolved it.
it appears open jdk add two extra columns to the output of the alternatives call. i belive it is setting the --family when it installs the alternative.
example:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/jre/bin/java - family java-1.8.0-openjdk.x86_64 priority 1800111
Created 01-04-2017 12:50 PM
Indeed, you are correct about the columns.
Please see my recent posts in the following thread for some more information:
Mismatched CDH versions: host has NONE but role expect 5
I outlined a proposed fix (expiremental) if you really need the JDK. Cloudera engineering is reviewing to decide on how we will address it in future releases.
Cheers,
Ben
Created 08-18-2017 02:57 AM
Is the issue fixed in a recent cloudera manager releases?
We are running 5.8.3 and we are experiencing the issue not for a jdk but for a custom alternatives.
I'm asking because is simplier for us to just upgrade the cloudera manager than fixing the alternatives name...
Created 10-04-2017 06:53 AM
If anyone is interested i've updated to CM 5.12.1 and the issue has been fixed.
Created 06-14-2017 07:21 AM
Created 06-15-2017 03:52 AM
Created 11-22-2016 01:58 AM
Created 11-22-2016 02:03 AM
If this is related to OPenJDK you have to remove OPenJDK from your server. Otherwise you have find out which packages make a proble. To do that you can run previously mentioned commands. As a result of the second you will get the information of packages that make a problems.
I have a vanilla red hat installation so I just remove them with:
yum remove packageName
Created 11-22-2016 02:33 AM
Created 01-08-2017 09:29 PM
I had the same issue. After went through the alternative list, I found there are three more openjdk items, which are jre_openjdk, jre_1.8.0 and jre_1.8.0_openjdk. After I removed these three items, CM started distributing task.
Created 12-28-2016 10:31 AM
Created 04-08-2019 06:03 PM
Hi,
Could you please let me know how to solve this issue. I'm having the same error mentioned below.
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 710, in start
self._init_after_first_heartbeat_response(resp_data)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 840, in _init_after_first_heartbeat_response
self.client_configs.load()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 682, in load
new_deployed.update(self._lookup_alternatives(fname))
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 432, in _lookup_alternatives
return self._parse_alternatives(alt_name, out)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 444, in _parse_alternatives
path, _, _, priority_str = line.rstrip().split(" ")
ValueError: too many values to unpack
I tried to see the jdk versions. But same jdk is present in all other servers. Not sure why am having issue only with this server.
[root@ip-10-0-1-32 ~]# rpm -qa "*jdk*"
java-1.8.0-openjdk-devel-1.8.0.144-0.b01.el7_4.x86_64
java-1.8.0-openjdk-headless-1.8.0.144-0.b01.el7_4.x86_64
jdk1.8.0_102-1.8.0_102-fcs.x86_64
copy-jdk-configs-2.2-3.el7.noarch
java-1.8.0-openjdk-1.8.0.144-0.b01.el7_4.x86_64
Thanks