question Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector in Support Questions

HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

maurice_knopp — Thu, 09 May 2019 04:48:52 GMT

Hi Guys,

we are trying to install the HDP 3.1.0 platform on-premise.

We have finished the installation but ran into a problem setting up MariaDB for the Hive metastore.

Hive explicitly lists MariaDB as supported:

https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administration

and suggests using the driver class "org.mariadb.jdbc.Driver"

When trying to setup "JDBC Driver Class" to "org.mariadb.jdbc.Driver" in Ambari of the HDP3.1.0 installation, then the start of the metastore fails with:

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 60, in <module>
HiveClient().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 966, in restart
self.install(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 38, in install
import params
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/params.py", line 273, in <module>
else: raise Fail(format("JDBC driver '{hive_jdbc_driver}' not supported."))
resource_management.core.exceptions.Fail: JDBC driver 'org.mariadb.jdbc.Driver' not supported.

As far as I can tell, the driver class "org.mariadb.jdbc.Driver" is not included in the role out script.

Can you help us or is there a fix for this?

Thanks!

Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

Shelton — Sat, 17 Aug 2019 22:33:17 GMT

@Maurice Knopp

Maria DB uses the MySQL drivers, you will need to the below steps to successfully deploy hive, this is my HDP 3.1 installation so I am pretty sure of what I am saying.

The Hortonworks matrix helps you choose the right versions for compatibility for HDP 3.1.0 you will need MariaDB 10.2 or MySQL 5.7 only with InnoDB

Ensure that you change the JDBC drivers as shown in the screen

# yum install -y mysql-connector-java

Make ambari aware of the driver

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

The above setup won't destroy your Ambari installation!

Please let me know

Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

maurice_knopp — Fri, 10 May 2019 15:51:33 GMT

We were not able to solve the problem yet. Currently, we are waiting for professional consulting support.

I will post any furhter details on that issue here when we get more information about the cause of the problem.

Thank you for your reply!

Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

Shelton — Fri, 10 May 2019 16:32:32 GMT

@Maurice Knopp

You could save your company money 🙂 This is a trivial matter can you share your HS2 logs? What OS's are running on?

Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

maurice_knopp — Fri, 10 May 2019 20:36:43 GMT

Of course we want to save money 😉

We thought the database issue is the reason we have problems connecting to the hive metastore but we now figured out that the original source for this issue seems to be related to kerberos authentication.

The main problem is, that our (new) HDP 3.1 Cluster is not running the hive scripts we developed on our "old" HDP 2.3 installation and we got lots of vertex failures.

I will try to provide more information with the help of our admin.

We are using CentOS 7.6 (all recent updates installed).

Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

Shelton — Sat, 11 May 2019 00:28:52 GMT

@Maurice Knopp

Great news if you are on Centos and kerberized environment I am just wondering how the Kerberos have you tried regenerating the specific keytab?

About you hql scripts you should realize there has been a lot of changes between HDP 2.3 with hive 1.2.1 and HDP 3.1 with hive version 3.1.0

About your MariaDB database and running the below version with HDP 3.1

"resource_management.core.exceptions.Fail: JDBC driver 'org.mariadb.jdbc.Driver' not supported."

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 30
Server version: 5.5.60-MariaDB MariaDB Serv
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and other
Type 'help;' or '\h' for help. Type '\c' to clear the current input statemen
MariaDB [(none)]>

Please did you run the steps I mentioned in http://community.hortonworks.com/answers/245833/view.html

Please revert

Re: HDP3.1.0 + MariaDB: Problem deploying MariaDB connector

maurice_knopp — Mon, 20 May 2019 15:54:10 GMT

Just for further reference: our issues was solved here

https://community.hortonworks.com/questions/246302/hive-tez-vertex-failed-error-during-reduce-phase-h.html