Support Questions

vibe · ‎11-27-2017

Hi,

After adding the impala daemon role to the master node, it fails to start. It starts fine on the 5 compute nodes. Please help point me in the right direction.

Thanks,

Vibe

3:06:26.292 PM

INFO

cc:122

Invalid short-circuit reads configuration:
  - Impala cannot read or execute the parent directory of dfs.domain.socket.path

    @           0x83da19  impala::Status::Status()
    @           0xaa3df6  impala::Frontend::ValidateSettings()
    @           0xaca3b0  impala::ImpalaServer::ImpalaServer()
    @           0xacb978  impala::CreateImpalaServer()
    @           0xb3ef28  ImpaladMain()
    @           0x7d96a3  main
    @     0x7f8bd4442c05  __libc_start_main
    @           0x80c65d  (unknown)

3:06:26.292 PM

ERROR

cc:268

Invalid short-circuit reads configuration:
  - Impala cannot read or execute the parent directory of dfs.domain.socket.path

3:06:26.292 PM

ERROR

cc:271

Aborting Impala Server startup due to improper configuration. Impalad exiting.

vibe · ‎11-27-2017

Additional info:

Can't open /run/cloudera-scm-agent/process/513-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/513-impala-IMPALAD/proc.json: Permission denied.

csguna · ‎11-27-2017

are you managing your cluster using cloudera manager ?

what version of CDH ?

what version of CM ?

Please provide the full stack trace of impala logs .

also please use the below tag for pasting the logs for more readability

Insert code - icon for logs / error message for more readability

csguna · ‎11-27-2017

Are you mananing the cluster using cloudera manager ?

Looks like you have not enabled short circuit read in the hdfs level and as well as client

This link will for sure help you fix this issue

https://www.cloudera.com/documentation/enterprise/5-3-x/topics/admin_hdfs_short_circuit_reads.html

check if the below tags and its corresponding value reflect in hdfs-site.xml

<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>

<property>
    <name>dfs.client.read.shortcircuit.streams.cache.size</name>
    <value>1000</value>
</property>


<property>
    <name>dfs.client.read.shortcircuit.streams.cache.expiry.ms</name>
    <value>10000</value>
</property>

<property>
    <name>dfs.domain.socket.path</name>
    <value>/var/run/hadoop-hdfs/dn._PORT</value>
</property
>

var/run/hadoop-hdfs/ is group-writable, make sure its group is root.

Please let me know if you need more information.

vibe · ‎11-28-2017

Hello,

Thank you for the replay. I'm using Cloudera Express 5.12.1 and have verified Enable HDFS Short-Circuit Read is checked. How do I provide you with a stack trace?

vibe · ‎11-28-2017

Is this what you're requesting?

Tue Nov 28 09:43:45 PST 2017
++ hostname -f
++ hostname -i
+ echo 'Running on: server'
++ dirname /usr/lib64/cmf/service/impala/impala.sh
+ cloudera_config=/usr/lib64/cmf/service/impala
++ cd /usr/lib64/cmf/service/impala/../common
++ pwd
+ cloudera_config=/usr/lib64/cmf/service/common
+ . /usr/lib64/cmf/service/common/cloudera-config.sh
++ set -x
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/meta/cdh_env.sh ']'
+ OLD_IFS=' 	
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS=' 	
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.12.1-1.cdh5.12.1.p0.3
+ . /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/meta/cdh_env.sh
++ CDH_DIRNAME=CDH-5.12.1-1.cdh5.12.1.p0.3
++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-0.20-mapreduce
++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-0.20-mapreduce
++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-hdfs
++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-hdfs
++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-httpfs
++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-httpfs
++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-mapreduce
++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-mapreduce
++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-yarn
++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-yarn
++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase
++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase
++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/zookeeper
++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/zookeeper
++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive
++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive
++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hue
++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hue
++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/oozie
++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/oozie
++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop
++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/flume-ng
++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/flume-ng
++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/pig
++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/pig
++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive-hcatalog
++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hive-hcatalog
++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sqoop2
++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/llama
++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/llama
++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sentry
++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/sentry
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/bin/hadoop
++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/solr
++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/solr
++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase-solr
++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase-solr
++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/search
++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/search
++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/spark
++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/spark
++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-kms
++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-kms
++ export CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/parquet
++ CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/parquet
++ export CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/avro
++ CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/avro
+ export IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
+ IMPALA_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala
+ export IMPALA_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf
+ IMPALA_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf
+ export HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hadoop-conf
+ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hadoop-conf
+ export HIVE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hive-conf
+ HIVE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hive-conf
+ export HBASE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hbase-conf
+ HBASE_CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD/hbase-conf
++ replace_pid
++ echo
++ sed 's#{{PID}}#11676#g'
+ export JAVA_TOOL_OPTIONS=
+ JAVA_TOOL_OPTIONS=
+ [[ -d /run/cloudera-scm-agent/process/532-impala-IMPALAD/hbase-conf ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -z '' ]]
+ export AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ AUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -z '' ]]
+ export CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -n '' ]]
+ FLAG_FILE=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
+ USE_DEBUG_BUILD=false
+ replace_conf_dir
+ echo CONF_DIR=/run/cloudera-scm-agent/process/532-impala-IMPALAD
+ echo CMF_CONF_DIR=/etc/cloudera-scm-agent
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'httpfs.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh httpfs.sh hue.sh impala.sh sqoop.sh supervisor.conf '*.log' impala.keytab '*jceks'
+ find /run/cloudera-scm-agent/process/532-impala-IMPALAD -type f '!' -path '/run/cloudera-scm-agent/process/532-impala-IMPALAD/logs/*' '!' -name cloudera-config.sh '!' -name httpfs.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name '*.log' '!' -name impala.keytab '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/532-impala-IMPALAD#g' '{}' ';'
Can't open /run/cloudera-scm-agent/process/532-impala-IMPALAD/config.zip: Permission denied.
Can't open /run/cloudera-scm-agent/process/532-impala-IMPALAD/proc.json: Permission denied.
+ make_scripts_executable
+ find /run/cloudera-scm-agent/process/532-impala-IMPALAD -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/532-impala-IMPALAD#g' /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
+ perl -pi -e 's#{{CGROUP_ROOT_CPU}}#/sys/fs/cgroup/cpu,cpuacct#g' /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
+ '[' -f /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/.htpasswd ']'
+ chmod 600 /run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/.htpasswd
+ false
+ export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala/sbin-retail
+ IMPALA_BIN=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala/sbin-retail
+ [[ true = '' ]]
+ '[' impalad = impalad ']'
+ exec /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/532-impala-IMPALAD/impala-conf/impalad_flags
Redirecting stderr to /var/log/impalad/impalad.ERROR

Support Questions

Failed to start impala daemon on host (master node) after adding role