question Re: Hive DB issue in connection in Support Questions

Hive DB issue in connection

irfangk1 — Mon, 07 Oct 2019 14:33:57 GMT

Hi,

Below is the logs

Host checks started.
Check execute list: db_connection_check
DB connection check started.
WARNING: File /var/lib/ambari-agent/cache/DBConnectionVerification.jar already exists, assuming it was downloaded before
WARNING: File /var/lib/ambari-agent/cache/mysql-connector-java.jar already exists, assuming it was downloaded before
call['/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-agent/cache/DBConnectionVerification.jar:/var/lib/ambari-agent/cache/mysql-connector-java.jar -Djava.library.path=/var/lib/ambari-agent/cache org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://host.com/metastore" "hive" [PROTECTED] com.mysql.jdbc.Driver'] {}
call returned (1, 'ERROR: Unable to connect to the DB. Please check DB connection properties.\ncom.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.')
DB connection check completed.
Host checks completed.
Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Command failed after 1 tries

Re: Hive DB issue in connection

Shelton — Mon, 07 Oct 2019 09:16:48 GMT

@irfangk1

Can you share your ambari.server. properties and your /etc/hosts

Re: Hive DB issue in connection

irfangk1 — Wed, 09 Oct 2019 05:54:22 GMT

ambari.server. properties as below

agent.package.install.task.timeout=1800
agent.stack.retry.on_repo_unavailability=false
agent.stack.retry.tries=5
agent.task.timeout=900
agent.threadpool.size.max=25
ambari-server.user=root
ambari.python.wrap=ambari-python-wrap
bootstrap.dir=/var/run/ambari-server/bootstrap
bootstrap.script=/usr/lib/ambari-server/lib/ambari_server/bootstrap.py
bootstrap.setup_agent.script=/usr/lib/ambari-server/lib/ambari_server/setupAgent .py
check_database_skipped=false
client.threadpool.size.max=25
common.services.path=/var/lib/ambari-server/resources/common-services
custom.action.definitions=/var/lib/ambari-server/resources/custom_action_definit ions
custom.mysql.jdbc.name=mysql-connector-java.jar
custom.postgres.jdbc.name=postgresql-42.2.8.jar
extensions.path=/var/lib/ambari-server/resources/extensions
gpl.license.accepted=false
http.cache-control=no-store
http.charset=utf-8
http.pragma=no-cache
http.strict-transport-security=max-age=31536000
http.x-content-type-options=nosniff
http.x-frame-options=DENY
http.x-xss-protection=1; mode=block
java.home=/usr/jdk64/jdk1.8.0_112
java.releases=jdk1.8
java.releases.ppc64le=
jce.download.supported=true
jce.name=jce_policy-8.zip
jdk.download.supported=true
jdk.name=jdk-8u112-linux-x64.tar.gz
jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
jdk1.8.dest-file=jdk-8u112-linux-x64.tar.gz
jdk1.8.home=/usr/jdk64/
jdk1.8.jcpol-file=jce_policy-8.zip
jdk1.8.jcpol-url=http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip
jdk1.8.re=(jdk.*)/jre
jdk1.8.url=http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.ta r.gz
kerberos.keytab.cache.dir=/var/lib/ambari-server/data/cache
kerberos.operation.verify.kdc.trust=true
local.database.user=postgres
metadata.path=/var/lib/ambari-server/resources/stacks
mpacks.staging.path=/var/lib/ambari-server/resources/mpacks
pid.dir=/var/run/ambari-server
previous.custom.mysql.jdbc.name=mysql-connector-java.jar
recommendations.artifacts.lifetime=1w
recommendations.dir=/var/run/ambari-server/stack-recommendations
resources.dir=/var/lib/ambari-server/resources
rolling.upgrade.skip.packages.prefixes=
security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECD HE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WI TH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_ 256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_ CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH _ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_A ES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC _SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TL S_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DH E_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDS A_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_C BC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS _DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_W ITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DE S_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_S HA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_an on_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_A ES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CB C_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_D H_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_S HA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH _DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES 40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_EC DHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_EC DH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SS L_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD 5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_C BC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5
security.server.keys_dir=/var/lib/ambari-server/keys
server.connection.max.idle.millis=900000
server.execution.scheduler.isClustered=false
server.execution.scheduler.maxDbConnections=5
server.execution.scheduler.maxThreads=5
server.execution.scheduler.misfire.toleration.minutes=480
server.fqdn.service.url=http://169.254.169.254/latest/meta-data/public-hostname
server.http.session.inactive_timeout=1800
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
server.os_family=ubuntu16
server.os_type=ubuntu16
server.persistence.type=local
server.python.log.level=INFO
server.python.log.name=ambari-server-command.log
server.stages.parallel=true
server.task.timeout=1200
server.tmp.dir=/var/lib/ambari-server/data/tmp
server.version.file=/var/lib/ambari-server/resources/version
shared.resources.dir=/usr/lib/ambari-server/lib/ambari_commons/resources
skip.service.checks=false
stack.java.home=/usr/jdk64/jdk1.8.0_112
stack.jce.name=jce_policy-8.zip
stack.jdk.name=jdk-8u112-linux-x64.tar.gz
stackadvisor.script=/var/lib/ambari-server/resources/scripts/stack_advisor.py
ulimit.open.files=65536
upgrade.parameter.convert.hive.tables.timeout=86400
upgrade.parameter.move.hive.tables.timeout=86400
user.inactivity.timeout.default=0
user.inactivity.timeout.role.readonly.default=0
views.ambari.request.connect.timeout.millis=30000
views.ambari.request.read.timeout.millis=45000
views.http.cache-control=no-store
views.http.charset=utf-8
views.http.pragma=no-cache
views.http.strict-transport-security=max-age=31536000
views.http.x-content-type-options=nosniff
views.http.x-frame-options=SAMEORIGIN
views.http.x-xss-protection=1; mode=block
views.request.connect.timeout.millis=5000
views.request.read.timeout.millis=10000
views.skip.home-directory-check.file-system.list=wasb,adls,adl
webapp.dir=/usr/lib/ambari-server/web

*** etc/host below
127.0.0.1 localhost
127.0.0.1 inairsr541012v9

172.27.3.12 inairsr541012v9.ntil.com inairsr541012v9
172.27.3.13 inairsr542007v3.ntil.com inairsr542007v3
172.27.3.14 inairsr542007v4.ntil.com inairsr542007v4

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Re: Hive DB issue in connection

Shelton — Wed, 09 Oct 2019 06:23:45 GMT

@irfangk1

There is something wrong I don't see the database entry in your ambari.server.properties how will it bind ?
Something like below

server.jdbc.url=jdbc:postgresql://<HOSTNAME>:<PORT>/ambari?ssl=true

Can you validate, you are using the embedded postgres right?

Re: Hive DB issue in connection

irfangk1 — Wed, 09 Oct 2019 06:35:42 GMT

Previously i was with Postgresql, now i am on mysql.. can you Please share the solution for the connection

Re: Hive DB issue in connection

irfangk1 — Thu, 10 Oct 2019 04:40:28 GMT

Hi Shelton,

I am getting below error in mysql, & please share how to deal with Connection Failure

am not able to start mysql.. below are the error..

Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.

Re: Hive DB issue in connection

Shelton — Thu, 10 Oct 2019 08:22:19 GMT

@irfangk1

There are a couple of things to do first check your disk space and inode usage.
To rule out permissions, there a are the listings for the relevant directory:
$ ls -la /var/lib/mysql

Can you share /var/log/mysql/mysql.log

What the value of innodb-buffer-pool size in the config file /etc/mysql/my.cnf can you edit the my.cnf by adding

[mysqld]
innodb_force_recovery = 1

And then running:

sudo systemctl start mysql

Hope that helps

Re: Hive DB issue in connection

irfangk1 — Thu, 10 Oct 2019 08:46:42 GMT

drwxr-x--- 2 mysql mysql 4096 Oct 3 09:28 mysql Permission

disk usage is below 5% and

inode Below

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 2045745 426 2045319 1% /dev
tmpfs 2053300 957 2052343 1% /run
/dev/sda1 6496256 299565 6196691 5% /
tmpfs 2053300 7 2053293 1% /dev/shm
tmpfs 2053300 5 2053295 1% /run/lock
tmpfs 2053300 17 2053283 1% /sys/fs/cgroup
tmpfs 2053300 18 2053282 1% /run/user/108
tmpfs 2053300 4 2053296 1% /run/user/0
tmpfs 2053300 4 2053296 1% /run/user/1009
tmpfs 2053300 4 2053296 1% /run/user/1010
tmpfs 2053300 4 2053296 1% /run/user/1006
tmpfs 2053300 4 2053296 1% /run/user/1004

-- myconf file below
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#

One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
innodb_force_recovery = 1
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 172.27.3.12

# * Fine Tuning
#
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

Re: Hive DB issue in connection

Shelton — Thu, 10 Oct 2019 20:01:09 GMT

@irfangk1

This line in your error log says it all

org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://host.com/metastore" "hive"

The format should be jdbc:mysql://host-name: port usually for Mysql it's 3306 to initiate any database connectivity you need to know the port at which the database is waiting for connections

Ambari.properties

# cat /etc/ambari-server/conf/ambari.properties | grep mysql
custom.mysql.jdbc.name=mysql-connector-java.jar
previous.custom.mysql.jdbc.name=mysql-connector-java.jar
server.jdbc.database=mysql
server.jdbc.driver=com.mysql.jdbc.Driver
server.jdbc.rca.driver=com.mysql.jdbc.Driver
server.jdbc.rca.url=jdbc:mysql://your_host:3306/ambari
server.jdbc.url=jdbc:mysql://your_host:3306/ambari

Backup these files

Verification jar

# mv /var/lib/ambari-agent/cache/DBConnectionVerification.jar /var/lib/ambari-agent/cache/DBConnectionVerification.jar.bck

MySql connector

# mv /var/lib/ambari-agent/cache/mysql-connector-java.jar /var/lib/ambari-agent/cache/mysql-connector-java.jar.bck

Then restart Ambari and the agent

Re: Hive DB issue in connection

Shelton — Thu, 10 Oct 2019 20:18:23 GMT

@irfangk1

If you switched to Mysql why do you have these entries in your ambari.properties

server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari

Can you share the output of

# cat /etc/ambari-server/conf/ambari.properties | grep mysql

Why is that in the log again I see "jdbc:mysql://host.com/metastore" "hive" where is the PORT number?

Re: Hive DB issue in connection

irfangk1 — Fri, 11 Oct 2019 03:56:47 GMT

cat /etc/ambari-server/conf/ambari.properties | grep mysql
custom.mysql.jdbc.name=mysql-connector-java.jar
previous.custom.mysql.jdbc.name=mysql-connector-java.jar

Re: Hive DB issue in connection

Shelton — Fri, 11 Oct 2019 04:19:26 GMT

@irfangk1
Can you share the screenshot of your Ambari Hive config because when I see "jdbc:mysql://host.com/metastore" "hive" that means you dont have not The default TCP port

for PostgreSQL is usually 5432 see in the postgresql.conf

Re: Hive DB issue in connection

irfangk1 — Fri, 11 Oct 2019 05:00:27 GMT

Mysql is not starting now check the below error

service mysql restart
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.

& after putting port of mysql,

WARNING: File /var/lib/ambari-agent/cache/DBConnectionVerification.jar already exists, assuming it was downloaded before
WARNING: File /var/lib/ambari-agent/cache/mysql-connector-java.jar already exists, assuming it was downloaded before
2019-10-11 10:26:28,466 - call['/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-agent/cache/DBConnectionVerification.jar:/var/lib/ambari-agent/cache/mysql-connector-java.jar -Djava.library.path=/var/lib/ambari-agent/cache org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://172.27.3.12: 3306" "hive" [PROTECTED] com.mysql.jdbc.Driver'] {}
2019-10-11 10:26:28,607 - call returned (1, 'ERROR: Unable to connect to the DB. Please check DB connection properties.\ncom.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Cannot load connection class because of underlying exception: \'java.lang.NumberFormatException: For input string: " 3306"\'.')
2019-10-11 10:26:28,608 - DB connection check completed.
2019-10-11 10:26:28,608 - Host checks completed.
2019-10-11 10:26:28,608 - Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Cannot load connection class because of underlying exception: 'java.lang.NumberFormatException: For input string: " 3306"'.

Command failed after 1 tries

Re: Hive DB issue in connection

Shelton — Fri, 11 Oct 2019 06:53:10 GMT

@irfangk1

That means you didn't install MySQL. Can I ask you one question? What documentation are you using as a reference?The basic steps are described when you are preparing the environment. To me it seems you launched the deployment using the embedded database Postgresql but now you want to switch to MySQL. To do that you need to install mysql and pre-create the ambari database before launching the Ambari server setup, that explains why you are getting all those errors.
I can send you a procedure in 9 hours

Re: Hive DB issue in connection

irfangk1 — Fri, 11 Oct 2019 09:24:18 GMT

Please send procedure. Thanks

Re: Hive DB issue in connection

irfangk1 — Sun, 13 Oct 2019 03:49:25 GMT

Hi Shelton,

I am waiting for the procedure

Re: Hive DB issue in connection

Shelton — Sun, 13 Oct 2019 08:38:24 GMT

@irfangk1

Please download it from here irfangk1_download

HTH