Failed to start SAM and schema registry service in HDF-3.3.0.0-165

[#1]

Description

HDF: Fails to start Schema Registry and Streaming, but overall cluster status returns 'ready'

1. Create HDF3.3 with local repo (1 x master and 2 x worker)

Streaming Analytics Manager log

2020-03-03 22:29:13,456 - Making http requests to http://bluedata-18.bdlocal:7777/api/v1/config/streamline
2020-03-03 22:29:13,457 - Failed to make API request to SAM server at http://bluedata-18.bdlocal:7777/api/v1/config/streamline,retrying.. 1 out of 6
2020-03-03 22:29:18,462 - Making http requests to http://bluedata-18.bdlocal:7777/api/v1/config/streamline
2020-03-03 22:29:18,463 - Failed to make API request to SAM server at http://bluedata-18.bdlocal:7777/api/v1/config/streamline,retrying.. 2 out of 6
2020-03-03 22:29:28,473 - Making http requests to http://bluedata-18.bdlocal:7777/api/v1/config/streamline
2020-03-03 22:29:28,474 - Failed to make API request to SAM server at http://bluedata-18.bdlocal:7777/api/v1/config/streamline,retrying.. 3 out of 6
2020-03-03 22:29:43,489 - Making http requests to http://bluedata-18.bdlocal:7777/api/v1/config/streamline
2020-03-03 22:29:43,490 - Failed to make API request to SAM server at http://bluedata-18.bdlocal:7777/api/v1/config/streamline,retrying.. 4 out of 6
2020-03-03 22:30:03,499 - Making http requests to http://bluedata-18.bdlocal:7777/api/v1/config/streamline
2020-03-03 22:30:03,499 - Failed to make API request to SAM server at http://bluedata-18.bdlocal:7777/api/v1/config/streamline,retrying.. 5 out of 6
2020-03-03 22:30:28,524 - Failed to make API request to SAM server at http://bluedata-18.bdlocal:7777/api/v1/config/streamline after 6
2020-03-03 22:30:28,525 - Execute['find /var/log/streamline -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'streamline'}
==> /var/log/streamline/streamline.err <==
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource
at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:93)
at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:319)
at com.zaxxer.hikari.pool.PoolBase.<init>(PoolBase.java:114)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.prepare(HikariCPConnectionBuilder.java:49)
at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.<init>(HikariCPConnectionBuilder.java:38)
at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.getHikariCPConnnectionBuilder(QueryExecutorFactory.java:79)
at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.get(QueryExecutorFactory.java:45)
at com.hortonworks.registries.storage.impl.jdbc.JdbcStorageManager.init(JdbcStorageManager.java:240)
at com.hortonworks.streamline.webservice.StreamlineApplication.getStorageManager(StreamlineApplication.java:194)
at com.hortonworks.streamline.webservice.StreamlineApplication.getDao(StreamlineApplication.java:183)
at com.hortonworks.streamline.webservice.StreamlineApplication.registerResources(StreamlineApplication.java:215)
at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:101)
at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:75)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
at io.dropwizard.cli.Cli.run(Cli.java:75)
at io.dropwizard.Application.run(Application.java:79)
at com.hortonworks.streamline.webservice.StreamlineApplication.main(StreamlineApplication.java:79)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:80)
... 19 more

I am using mysql as Database.

[root@bluedata-17 ~]# mysql --version
mysql Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using EditLine wrapper

Can anyone help me on this? Any Suggestion?

Thanks

[#2]

@Vikas_Samant   looks like this might be the issue

Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource

Did you install the mysql JDBC?  In a demo yesterday (centos/rhel) that looks like:

yum install mysql-connector-java
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

[#3]

@stevenmatison thanks.for your reply 

I checked in my set up,  mysql JDBC is already setup. I run both the command you provided, it was already there. Actually i have 2 HDF cluster. 1st cluster HDF cluster is in ready state with all services online. In 2nd HDF cluster schema registry and SAM services are failing because of the above error. While troubleshooting i saw one minor difference in mysql version in both the setup.

1st HDF cluster mysql version --

[root@bluedata-342 bluedata]# cd
[root@bluedata-342 ~]# mysql --version
mysql Ver 14.14 Distrib 5.7.28, for Linux (x86_64) using EditLine wrapper
[root@bluedata-342 ~]#

2nd HDF cluster mysql version --

[root@bluedata-17 ~]#
[root@bluedata-17 ~]# mysql --version
mysql Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using EditLine wrapper
[root@bluedata-17 ~]#

Is this possible, it is creating issue?

[#4]

From what I've investigated, it may be due to the version of the mysql-connector-java.

The class com.mysql.jdbc.jdbc2.optional.MysqlDataSource is present in version 5 and the one that I have installed is the 8.

I tried installing explicitly the version 5 but I got different errors.

What I've done and got it working is changing that class to the one available in version 8, which is com.mysql.cj.jdbc.MysqlXADataSource.

For Schema Registry you have to modify 2 files:

• /var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.4.1.1-4/common-services/REGISTRY/0.3.0/package/scripts/params.py
• /var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/params.py

There change the variable registry_storage_java_class to the value com.mysql.cj.jdbc.MysqlXADataSource

Note: the variable should appear twice.

The same for Streaming Analytics, you have to modify 2 files. This time:

• /var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.4.1.1-4/common-services/STREAMLINE/0.5.0/package/scripts/params.py
• /var/lib/ambari-agent/cache/common-services/STREAMLINE/0.5.0/package/scripts/params.py

There change the variable streamline_storage_java_class also to the value com.mysql.cj.jdbc.MysqlXADataSource

You should then be able to start the services.

I just solved this, so I'm not aware if any other errors will show up by using these services.