Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

understanding HIVE METASTORE setup in HDP

understanding HIVE METASTORE setup in HDP

Master Collaborator

if I choose option "create new MySQL database" in configuring Metastore ,does it install and configure myql database on its own and I don't have to start the mysqld service on the host?

and does it run the MySQL_secure_installation script too ?

can you please specify whats the standard way of installing HIVE as I cant seem to understand it and everytime I install a cluster I get issues starting hive .

8 REPLIES 8
Highlighted

Re: understanding HIVE METASTORE setup in HDP

Mentor

@Sami Ahmad

Usually its good to set the hive user and database before launch the cluster setup, so that during the cluster hive configuration all is set !

##################################################
# Hive user and database setup
##################################################
mysql -u root -p [root_password]
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive_password';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'hive_host_name';
FLUSH PRIVILEGES;
##################################################
# Create the Hive database.
# The Hive database must be created before loading the Hive database schema.
##################################################
mysql -u hive -p[hive_password]
CREATE DATABASE hive;

Here is a hortonworks link

The other opton is creating the user and database on the fly as root. But you must run the mysql-connector-java.jar as part of the Ambari setup

Highlighted

Re: understanding HIVE METASTORE setup in HDP

Master Collaborator

hi Geoffrey

if i create the database and user as you instructed , before the hive config , then do i still have to run MySQL-connecto-java.jar ?

also I am not sure about the role of this command , when does one runs it ? i remember running it once to fix hive issues long ago on advise of forum users .

$HIVE_HOME/bin/schematool -initSchema -dbType mysql
Highlighted

Re: understanding HIVE METASTORE setup in HDP

Master Collaborator

also i created the hive user as per your instruction and i can connect to it on Linux box but the Ambari "Test connection" is failing with the following error, the proxy server is working fine and i can also ssh from each node to the other. The MySQL is installed on hadoop2 and ambari is running on hadoop1.

2017-06-16 10:59:13,101 - Error downloading DBConnectionVerification.jar from Ambari Server resources. Check network access to Ambari Server.
HTTP Error 504: Gateway Timeout
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 395, in execute_db_connection_check
    download_file(check_db_connection_url, check_db_connection_path)
  File "/usr/lib/python2.6/site-packages/ambari_commons/inet_utils.py", line 57, in download_file
    force_download_file(link, destination, chunk_size, progress_func = progress_func)
  File "/usr/lib/python2.6/site-packages/ambari_commons/inet_utils.py", line 125, in force_download_file
    response = urllib2.urlopen(request)
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 504: Gateway Timeout
2017-06-16 10:59:13,103 - Check db_connection_check was unsuccessful. Exit code: 1. Message: Error downloading DBConnectionVerification.jar from Ambari Server resources. Check network access to Ambari Server.
HTTP Error 504: Gateway Timeout

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 506, in <module>
    CheckHost().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 285, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 206, in actionexecute
    raise Fail(error_message)
resource_management.core.exceptions.Fail: Check db_connection_check was unsuccessful. Exit code: 1. Message: Error downloading DBConnectionVerification.jar from Ambari Server resources. Check network access to Ambari Server.
HTTP Error 504: Gateway Timeout
Highlighted

Re: understanding HIVE METASTORE setup in HDP

Super Mentor

@Sami Ahmad

I think you should really check with your Network Admin regarding the error

HTTP Error 504: Gateway Timeout

.

This indicates the issue of proxy configuration. I see that in past few days you have reported couple of other HCC Threads where we see that the root cause of all those issue is same "HTTP Error 504: Gateway Timeout" (with yum.conf while downloading packages sometimes while downloading jars from ambari server like in above case it is following error:

Error downloading DBConnectionVerification.jar from Ambari Server resources. Check network access to Ambari Server.
HTTP Error 504: Gateway Timeout

.

At ambari level you can try setting the HTTP proxy server information to the following script: /var/lib/ambari-server/ambari-env.sh. Please see: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.0/bk_ambari-reference/content/ch_setting_up_a...

-Dhttp.proxyHost=<yourProxyHost> -Dhttp.proxyPort=<yourProxyPort>

However, you need to make sure that your proxy server hostname & port are correct and your proxy server is running find without any issue.

If you continue to get the following error then i will suggest you to please check with your Network Admins.

HTTP Error 504: Gateway Timeout

.

.

For reference of other users who are getting the same error i am posting other threads references here:

Your other issues with the same error reported recently:

https://community.hortonworks.com/questions/107325/installing-ambari-with-local-repository-and-no-in...

https://community.hortonworks.com/questions/108888/node-registration-failing.html#comment-108904

Highlighted

Re: understanding HIVE METASTORE setup in HDP

Master Collaborator

is there anyway i can test on the command level if ambari-proxy is working ?

i have put the proxy settings in the ambari-env.sh file .

Highlighted

Re: understanding HIVE METASTORE setup in HDP

Super Mentor

@Sami Ahmad

Those proxy settings are JVM specific ... So if they are added to the ambari JVM then it is supposed to work.

-Dhttp.proxyHost=<yourProxyHost>-Dhttp.proxyPort=<yourProxyPort>

.

So if you want to confirm if those parameters are taken up by ambari process in run time or not then you can find them in the Ambari Process information like:

# ps -ef | grep -i AmbariServer

.

Highlighted

Re: understanding HIVE METASTORE setup in HDP

Master Collaborator

yes i see the proxy settings in the ambar-server , so why i am facing proxy issues? the proxy must be working since yum is able to download all the external modules and files.

root     15676     1  2 Jun15 ?        00:15:44 /usr/jdk64/jdk1.8.0_77/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhttp.proxyHost=dotatofwproxy.tolls.dot.state.fl.us -Dhttp.proxyPort=8080 -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer
[root@hadoop1 ambari-server]#

Highlighted

Re: understanding HIVE METASTORE setup in HDP

Super Mentor

@Sami Ahmad

Yum uses the proxy settings from "/etc/yum.conf" file by default. It might not be true for other utilities.

Also please check "~/.base_profile" and "~/.profile" scripts define the proxy settings so that it will be applicable globally.

export http_proxy=http://dotatofwproxy.tolls.dot.state.fl.us:8080

.

Another example, please try to validate if your mentioned proxy host and port is working correctly or not by using plain "wget" as following:

wget  -e use_proxy=yes -e http_proxy=http://dotatofwproxy.tolls.dot.state.fl.us:8080   http://$AMBARI_SERVERHOST:8080/resources/DBConnectionVerification.jar

Don't have an account?
Coming from Hortonworks? Activate your account here