Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to start impala daemon on host (master node) after adding role

Failed to start impala daemon on host (master node) after adding role

Explorer

Hi,

 

After adding the impala daemon role to the master node, it fails to start.  It starts fine on the 5 compute nodes.  Please help point me in the right direction.

 

Thanks,

Vibe

 

3:06:26.292 PM INFO cc:122
Invalid short-circuit reads configuration:
  - Impala cannot read or execute the parent directory of dfs.domain.socket.path

    @           0x83da19  impala::Status::Status()
    @           0xaa3df6  impala::Frontend::ValidateSettings()
    @           0xaca3b0  impala::ImpalaServer::ImpalaServer()
    @           0xacb978  impala::CreateImpalaServer()
    @           0xb3ef28  ImpaladMain()
    @           0x7d96a3  main
    @     0x7f8bd4442c05  __libc_start_main
    @           0x80c65d  (unknown)
3:06:26.292 PM ERROR cc:268
Invalid short-circuit reads configuration:
  - Impala cannot read or execute the parent directory of dfs.domain.socket.path

3:06:26.292 PM ERROR cc:271
Aborting Impala Server startup due to improper configuration. Impalad exiting.
5 REPLIES 5

Re: Failed to start impala daemon on host (master node) after adding role

Explorer

Additional info:

 

Can't open /run/cloudera-scm-agent/process/513-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/513-impala-IMPALAD/proc.json: Permission denied.

Re: Failed to start impala daemon on host (master node) after adding role

Champion

are you managing your cluster using cloudera manager ? 

what version of CDH ? 

what version of CM ? 

 

Please provide the full stack trace of impala logs . 

 

also please use the below tag for pasting the logs for more readability 

 

insertCodeForLogs.pngInsert code - icon for logs / error message for more readability

Re: Failed to start impala daemon on host (master node) after adding role

Champion

Are you mananing the cluster using cloudera manager ? 

 

Looks like you have not enabled short circuit read in the hdfs level and as well as client 

This link will for sure help you fix this issue 

 

https://www.cloudera.com/documentation/enterprise/5-3-x/topics/admin_hdfs_short_circuit_reads.html

 

check if the below tags and its corresponding value reflect in hdfs-site.xml 

 

 

<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>

<property>
    <name>dfs.client.read.shortcircuit.streams.cache.size</name>
    <value>1000</value>
</property>


<property>
    <name>dfs.client.read.shortcircuit.streams.cache.expiry.ms</name>
    <value>10000</value>
</property>

<property>
    <name>dfs.domain.socket.path</name>
    <value>/var/run/hadoop-hdfs/dn._PORT</value>
</property
>

 var/run/hadoop-hdfs/ is group-writable, make sure its group is root.

 

Please let me know if you need more information. 

 

 

Re: Failed to start impala daemon on host (master node) after adding role

Explorer

Hello,

 

Thank you for the replay.  I'm using Cloudera Express 5.12.1 and have verified Enable HDFS Short-Circuit Read is checked.  How do I provide you with a stack trace?

Re: Failed to start impala daemon on host (master node) after adding role

Explorer

Is this what you're requesting?

 

Tue Nov 28 09:43:45 PST 2017
++ hostname -f
++ hostname -i
+ echo 'Running on: server'
++ dirname /usr/lib64/cmf/service/impala/impala.sh
+ cloudera_config=/usr/lib64/cmf/service/impala
++ cd /usr/lib64/cmf/service/impala/../common
++ pwd
+ cloudera_config=/usr/lib64/cmf/service/common
+ . /usr/lib64/cmf/service/common/cloudera-config.sh
++ set -x
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/meta/cdh_env.sh ']'
+ OLD_IFS=' 	
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS=' 	
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.12.1-1.cdh5.12.1.p0.3
+ . /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/meta/cdh_env.sh
++ CDH_DIRNAME=CDH-5.12.1-1.cdh5.12.1.p0.3
++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-0.20-mapreduce
++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-0.20-mapreduce
++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-hdfs
++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-hdfs
++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-httpfs
++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-httpfs
++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-mapreduce
++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-mapreduce
++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-yarn
++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-yarn
++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase
++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase
++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/zookeeper
++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/zookeeper
++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive
++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive
++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hue
++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hue
++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/oozie
++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/oozie
++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/flume-ng
++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/flume-ng
++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/pig
++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/pig
++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive-hcatalog
++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive-hcatalog
++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sqoop2
++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/llama
++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/llama
++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sentry
++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sentry
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/bin/hadoop
++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/solr
++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/solr
++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase-solr
++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase-solr
++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/search
++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/search
++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/spark
++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/spark
++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-kms
++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-kms
++ export CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/parquet
++ CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/parquet
++ export CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/avro
++ CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/avro
+ export IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
+ IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
+ export IMPALA_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf
+ IMPALA_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf
+ export HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hadoop-conf
+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hadoop-conf
+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hive-conf
+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hive-conf
+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hbase-conf
+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hbase-conf
++ replace_pid
++ echo
++ sed 's#{{PID}}#11676#g'
+ export JAVA_TOOL_OPTIONS=
+ JAVA_TOOL_OPTIONS=
+ [[ -d /run/cloudera-scm-agent/process/532-impala-IMPALAD/hbase-conf ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -z '' ]]
+ export AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -z '' ]]
+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -n '' ]]
+ FLAG_FILE=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
+ USE_DEBUG_BUILD=false
+ replace_conf_dir
+ echo CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD
+ echo CMF_CONF_DIR=/etc/cloudera-scm-agent
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'httpfs.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh httpfs.sh hue.sh impala.sh sqoop.sh supervisor.conf '*.log' impala.keytab '*jceks'
+ find /run/cloudera-scm-agent/process/532-impala-IMPALAD -type f '!' -path '/run/cloudera-scm-agent/process/532-impala-IMPALAD/logs/*' '!' -name cloudera-config.sh '!' -name httpfs.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name '*.log' '!' -name impala.keytab '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/532-impala-IMPALAD#g' '{}' ';'
Can't open /run/cloudera-scm-agent/process/532-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/532-impala-IMPALAD/proc.json: Permission denied.
+ make_scripts_executable
+ find /run/cloudera-scm-agent/process/532-impala-IMPALAD -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/532-impala-IMPALAD#g' /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
+ perl -pi -e 's#{{CGROUP_ROOT_CPU}}#/sys/fs/cgroup/cpu,cpuacct#g' /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
+ '[' -f /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/.htpasswd ']'
+ chmod 600 /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/.htpasswd
+ false
+ export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala/sbin-retail
+ IMPALA_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala/sbin-retail
+ [[ true = '' ]]
+ '[' impalad = impalad ']'
+ exec /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
Redirecting stderr to /var/log/impalad/impalad.ERROR