Unexpected error. Unable to verify database connection

[#1]

hello:
I have got a problem.
I want append a HUE service to CM,when i executed the connection test, there has a connected issue :'quot;Unexpected error. Unable to verify database connection';.

The enviroment:

Centos 7.5 CDH5.15.1

mysql5.7.23

jdk1.8.0_161

I read the error logs carefully,

the server log:

Spoiler

2018-10-15 11:37:45,959 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection' 2018-10-15 11:37:45,960 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 1 2018-10-15 11:37:45,960 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: self._setup(name) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup self._wrapped = Settings(settings_module) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module __import__(name) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/settings.py", line 326, in "PASSWORD" : desktop.conf.get_database_password(), File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/conf.py", line 1695, in get_database_password password = DATABASE.PASSWORD_SCRIPT.get() File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 154, in get return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 270, in get_value return self._coerce_type(raw_val, prefix) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 290, in _coerce_type return self.type(raw) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 721, in coerce_password_from_script raise subprocess.CalledProcessError(p.returncode, script) subprocess.CalledProcessError: Command '{{CMF_CONF_DIR}}/altscript.sh sec-2-password' returned non-zero exit status 127 2018-10-15 11:37:45,960 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 336(HueTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.

and the agent stderr.log says:

Spoiler

echo CMF_CONF_DIR=/opt/cm-5.15.1/etc/cloudera-scm-agent + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks') ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' creds.localjceks + find /opt/cm-5.15.1/run/cloudera-scm-agent/process/261-HUE-test-db-connection -type f '!' -path '/opt/cm-5.15.1/run/cloudera-scm-agent/process/261-HUE-test-db-connection/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name creds.localjceks -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/opt/cm-5.15.1/run/cloudera-scm-agent/process/261-HUE-test-db-connection#g' '{}' ';' find: ‘perl’: 没有那个文件或目录 find: ‘perl’: 没有那个文件或目录 find: ‘perl’: 没有那个文件或目录 + make_scripts_executable + find /opt/cm-5.15.1/run/cloudera-scm-agent/process/261-HUE-test-db-connection -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';' + '[' is_db_alive == beeswax_server ']' + set_classpath_in_var HADOOP_EXTRA_CLASSPATH_STRING + '[' -z HADOOP_EXTRA_CLASSPATH_STRING ']' + [[ -n /opt/cm-5.15.1/share/cmf ]] ++ find /opt/cm-5.15.1/share/cmf/lib/plugins -maxdepth 1 -name '*.jar' ++ tr '\n' : + ADD_TO_CP=/opt/cm-5.15.1/share/cmf/lib/plugins/event-publish-5.15.1-shaded.jar:/opt/cm-5.15.1/share/cmf/lib/plugins/tt-instrumentation-5.15.1.jar: + [[ -n '' ]] + eval 'OLD_VALUE=$HADOOP_EXTRA_CLASSPATH_STRING' ++ OLD_VALUE= + NEW_VALUE=/opt/cm-5.15.1/share/cmf/lib/plugins/event-publish-5.15.1-shaded.jar:/opt/cm-5.15.1/share/cmf/lib/plugins/tt-instrumentation-5.15.1.jar: + export HADOOP_EXTRA_CLASSPATH_STRING=/opt/cm-5.15.1/share/cmf/lib/plugins/event-publish-5.15.1-shaded.jar:/opt/cm-5.15.1/share/cmf/lib/plugins/tt-instrumentation-5.15.1.jar + HADOOP_EXTRA_CLASSPATH_STRING=/opt/cm-5.15.1/share/cmf/lib/plugins/event-publish-5.15.1-shaded.jar:/opt/cm-5.15.1/share/cmf/lib/plugins/tt-instrumentation-5.15.1.jar + HUE=/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/bin/hue + [[ is_db_alive == runcpserver ]] + [[ is_db_alive == kt_renewer ]] + '[' dumpdata = is_db_alive ']' + '[' syncdb = is_db_alive ']' + '[' ldaptest = is_db_alive ']' + exec /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/bin/hue is_db_alive [15/Oct/2018 11:37:45 +0000] settings DEBUG DESKTOP_DB_TEST_NAME SET: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/desktop-test.db [15/Oct/2018 11:37:45 +0000] settings DEBUG DESKTOP_DB_TEST_USER SET: hue_test /bin/sh: {{CMF_CONF_DIR}}/altscript.sh: 没有那个文件或目录 Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/bin/hue", line 12, in load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')() File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/manage_entry.py", line 143, in entry execute_from_command_line(sys.argv) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line utility.execute() File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command commands = get_commands() File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands apps = settings.INSTALLED_APPS File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__ self._setup(name) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup self._wrapped = Settings(settings_module) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module __import__(name) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/settings.py", line 326, in "PASSWORD" : desktop.conf.get_database_password(), File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/conf.py", line 1695, in get_database_password password = DATABASE.PASSWORD_SCRIPT.get() File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 154, in get return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 270, in get_value return self._coerce_type(raw_val, prefix) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 290, in _coerce_type return self.type(raw) File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 721, in coerce_password_from_script raise subprocess.CalledProcessError(p.returncode, script) subprocess.CalledProcessError: Command '{{CMF_CONF_DIR}}/altscript.sh sec-2-password' returned non-zero exit status 127

the log says the file or path not exist,then i copied the file to this path on master and slave node.But the problem has not been solved.

[root@master ~]# ls /opt/cm-5.15.1/etc/cloudera-scm-agent -l
-rwxr-xr-x  1 root         root          363 10月 15 13:50 altscript.sh
-rw-r--r--. 1 cloudera-scm cloudera-scm 8873 10月 14 20:32 config.ini
[root@master ~]# ls /opt/cm-5.15.1/share/cmf/bin/altscript.sh -l
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 363 7月  31 18:28 /opt/cm-5.15.1/share/cmf/bin/altscript.sh

[root@slave183 logs]# ls /opt/cm-5.15.1/etc/cloudera-scm-agent/altscript.sh -l
-rwxrwxrwx. 1 root root 359 10月 15 01:12 /opt/cm-5.15.1/etc/cloudera-scm-agent/altscript.sh
[root@slave183 logs]# ls /opt/cm-5.15.1/share/cmf/bin/altscript.sh -l
-rwxrwxrwx. 1 cloudera-scm cloudera-scm 363 10月 14 20:42 /opt/cm-5.15.1/share/cmf/bin/altscript.sh

It's been bothering me for a long time, How can I solve this?

[#2]

Is it sovled? i got same when installing 6.2 on Redhat 7.6

Is there any way to bypass it

[#3]

@NightAndDay ,

Can you provide the same information that was provided in the original post?  It is important for us to know what the error or stack trace is in your issue.

If you have exactly the same stack trace, then it appears the CMF_CONF_DIR is not being replaced properly.  If this is a Cloudera Manager manged cluster, then CM will store passwords in a Java JCEKS file in the Hue process directory. To see the latest Hue process directory, you can use the following:

ls /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep HUE_SERVER| tail -1`

When the agent signals the supervisor to start Hue, it will run a hue.sh script to massage the configuration, set variables, etc. and then start the server.  Part of that is doing a replace on hue.ini that will replace "{{CMF_CONF_DIR}}" with the full path to the Hue process directory.

This replacement should also address hue.ini where the path to the password command is.  For example,

[[database]]
engine=postgres
host=huedb.example
port=7432
user=hue
password_script=/var/run/cloudera-scm-agent/process/2517-hue-HUE_SERVER/altscript.sh sec-8-password
name=hue

In the original post, the error was:

Command '{{CMF_CONF_DIR}}/altscript.sh sec-2-password' returned non-zero exit status 127

The replace string is still there, so it appears that the replace did not work as expected.  Since the literal path does not exist, a 127 error would make sense.

I'd recommend sharing the stderr.log from your database test to see if there are any clues about what is happening in your case.

I am not sure how this is getting set:

echo CMF_CONF_DIR=/opt/cm-5.15.1/etc/cloudera-scm-agent

Did you install your Cloudera Manager via tarball method perhaps?