@Jitendra Yadav thanks to your example I went to recheck /usr/hdp/current/hive-server2/bin/hiveserver2 file. I cross checked it will a working staging environment I have, and what do you know... the following line in the end was missing: exec "${HIVE_HOME}/bin/hiveserver2.distro" "$@" I modified the file with the missing line and now the service is starting without a problem. Thanks again!!! ... View more

@Jitendra Yadav there is no output.. co console messages or anything. I checked /usr/hdp/current/hive-server2/bin/hiveserver2 content, its just setting different paths and jar files, i dont think its were the services is running from. ... View more

@Pardeep perfect, thats exactly what i was looking for. while going over the log, the intresting bit is the follows: 016-05-24 16:05:43,261 - File['/etc/security/limits.d/hive.conf'] {'content': Template('hive.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2016-05-24 16:05:43,261 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://HOSTNAME:8080/resources/DBConnectionVerification.jar'), 'mode': 0644} 2016-05-24 16:05:43,262 - Not downloading the file from http://HOSTNAME:8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/tmp/DBConnectionVerification.jar already exists 2016-05-24 16:05:43,263 - File['/var/lib/ambari-agent/tmp/start_hiveserver2_script'] {'content': Template('startHiveserver2.sh.j2'), 'mode': 0755} 2016-05-24 16:05:43,263 - Directory['/var/run/hive'] {'owner': 'hive', 'mode': 0755, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-05-24 16:05:43,264 - Directory['/var/log/hive'] {'owner': 'hive', 'mode': 0755, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-05-24 16:05:43,264 - Directory['/var/lib/hive'] {'owner': 'hive', 'mode': 0755, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-05-24 16:05:43,264 - Ranger admin not installed 2016-05-24 16:05:43,265 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://HOSTNAME:8020' | head -1'] {} 2016-05-24 16:05:47,240 - call returned (0, '') 2016-05-24 16:05:47,241 - Execute['/var/lib/ambari-agent/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.log /var/run/hive/hive-server.pid /usr/hdp/current/hive-server2/conf/conf.server /var/log/hive'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': '/usr/java/default', 'HADOOP_HOME': '/usr/hdp/current/hadoop-client'}, 'not_if': "ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p `ambari-sudo.sh su hive -l -s /bin/bash -c 'cat /var/run/hive/hive-server.pid'` >/dev/null 2>&1", 'user': 'hive', 'path': ['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/current/hadoop-client/bin']} 2016-05-24 16:05:47,372 - Execute['/usr/java/default/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-server2/lib/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://HOSTNAME/hive?createDatabaseIfNotExist=true' hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10} but I dont have any errors, and when checking /var/lib/ambari-agent/tmp/start_hiveserver2_script, the content is: HIVE_SERVER2_OPTS=" -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=$5" HIVE_CONF_DIR=$4 /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=" " ${HIVE_SERVER2_OPTS} > $1 2> $2 & echo $!|cat>$3 which translates to: HIVE_SERVER2_OPTS=" -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive" HIVE_CONF_DIR=/usr/hdp/current/hive-server2/conf/conf.server /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=" " -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive > /var/log/hive/hive-server2.out 2> /var/log/hive/hive-server2.log & echo $!|cat>$3 I tried running it manually and nothing happens, but I still nowhere where its calling the actual service itself, something like /usr/bin/hive --service hiveserver2. what am I missing here? ... View more

@Jitendra Yadav yes, I've already looked there and restarted ambari-agent and even ambari-server. I am checking for HiveServer2 logs (in /var/log/hive) but the problem is that the logs there are empty. for example the metastore logs are there and if I restart it the logs updates with information successfully. I know for a fact that its the right log location because when I restart HiveServer2 through ambari, the timestamp on those logs updates and corresponds to the time I try to start the service in ambari but the logs still show up as empty and don't provide any information. btw in the from the ambari-agent log: INFO 2016-05-24 16:04:21,115 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-24 16:04:21,180 Controller.py:260 - Heartbeat response received (id = 9) INFO 2016-05-24 16:04:21,181 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-24 16:04:31,183 Heartbeat.py:78 - Building Heartbeat: {responseId = 9, timestamp = 1464095071182, commandsInProgress = False, componentsMapped = True} INFO 2016-05-24 16:04:31,189 Controller.py:260 - Heartbeat response received (id = 10) INFO 2016-05-24 16:04:31,189 RecoveryManager.py:217 - HIVE_SERVER needs recovery hope that explains my situation. ... View more

Hi @Deepesh thanks for clarifying on how to start HiveServer2 manually. the problem is that I am checking for HiveServer2 logs (in /var/log/hive) but the problem is that the logs there are empty. for example the metastore logs are there and if I restart it the logs updates with information successfully. I know for a fact that its the right log location because when I restart HiveServer2 through ambari, the timestamp on those logs updates and corresponds to the time I try to start the service in ambari but the logs still show up as empty and don't provide any information. hope that explains my situation. Thanks. ... View more

@Sagar Shimpi i killed the process that was started manually and when i try to start hiveserver2 from ambari it does not start at all , thats why I opened this issue in the first place. I have the following error in ambari: Connection failed on host HOSTNAME:10000 (Execution of '! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2:// HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2:// HOSTNAME:10000/; transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)) ... View more

maybe this will help, but when starting hiveserver2 manually (/usr/bin/hive --service hiveserver2), I see its running with default memory values and not using the value in hive-site.xml, so maybe its why ambari having issues as well and i get the warning that i dont have heapsize set (16/05/23 14:06:54 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist): root 8622 30797 4 14:06 pts/0 00:00:09 /usr/java/default/bin/java -Xmx256m -Dhdp.version=2.3.2.0-2950 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.2.0-2950 -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.2.0-2950/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.2.0-2950/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.2.0-2950/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx256m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.2.0-2950/hive/lib/hive-service-1.2.1.2.3.2.0-2950.jar org.apache.hive.service.server.HiveServer2 ... View more

@Sagar Shimpi i restarted the ambari server and agent on the hive machine and the outcome is the same. this is the log from ambari-agent: INFO 2016-05-23 13:42:52,555 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-23 13:42:54,316 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-23 13:43:02,556 Heartbeat.py:78 - Building Heartbeat: {responseId = 29, timestamp = 1464000182555, commandsInProgress = False, componentsMapped = True} INFO 2016-05-23 13:43:02,577 Controller.py:260 - Heartbeat response received (id = 30) INFO 2016-05-23 13:43:02,577 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-23 13:43:12,578 Heartbeat.py:78 - Building Heartbeat: {responseId = 30, timestamp = 1464000192578, commandsInProgress = False, componentsMapped = True} INFO 2016-05-23 13:43:12,779 Controller.py:260 - Heartbeat response received (id = 31) INFO 2016-05-23 13:43:12,780 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-23 13:43:22,781 Heartbeat.py:78 - Building Heartbeat: {responseId = 31, timestamp = 1464000202781, commandsInProgress = False, componentsMapped = True} INFO 2016-05-23 13:43:22,825 Controller.py:260 - Heartbeat response received (id = 32) INFO 2016-05-23 13:43:22,826 RecoveryManager.py:217 - HIVE_SERVER needs recovery. WARNING 2016-05-23 13:43:29,367 base_alert.py:417 - [Alert][yarn_resourcemanager_webui] HA nameservice value is present but there are no aliases for {{yarn-site/yarn.resourcemanager.ha.rm-ids}} INFO 2016-05-23 13:43:29,393 logger.py:67 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/home', '/proc/sys/fs/binfmt_misc']. INFO 2016-05-23 13:43:29,397 logger.py:67 - Mount point for directory /hadoop/hdfs/data is / INFO 2016-05-23 13:43:29,408 logger.py:67 - Execute['export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://HOSTNAME:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;''] {'path': ['/bin/', '/usr/bin/', '/usr/sbin/', '/usr/hdp/current/hive-metastore/bin'], 'user': 'ambari-qa', 'timeout': 30} INFO 2016-05-23 13:43:29,425 logger.py:67 - Execute['! beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''] {'path': ['/bin/', '/usr/bin/', '/usr/lib/hive/bin/', '/usr/sbin/'], 'user': 'ambari-qa', 'timeout': 30} INFO 2016-05-23 13:43:32,826 Heartbeat.py:78 - Building Heartbeat: {responseId = 32, timestamp = 1464000212826, commandsInProgress = False, componentsMapped = True} INFO 2016-05-23 13:43:32,831 Controller.py:260 - Heartbeat response received (id = 33) INFO 2016-05-23 13:43:32,831 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-23 13:43:42,831 Heartbeat.py:78 - Building Heartbeat: {responseId = 33, timestamp = 1464000222831, commandsInProgress = False, componentsMapped = True} INFO 2016-05-23 13:43:42,875 Controller.py:260 - Heartbeat response received (id = 34) INFO 2016-05-23 13:43:42,875 RecoveryManager.py:217 - HIVE_SERVER needs recovery. INFO 2016-05-23 13:43:52,876 Heartbeat.py:78 - Building Heartbeat: {responseId = 34, timestamp = 1464000232875, commandsInProgress = False, componentsMapped = True} INFO 2016-05-23 13:43:52,944 Controller.py:260 - Heartbeat response received (id = 35) HDP version 2.3.2.0-2950 ambari version 2.1.2 ... View more

Hi @Sagar Shimpi , @Kuldeep Kulkarni some updates: I've started hiveserver2 like you mentioned: /usr/bin/hive --service hiveserver2 --hiveconf hive.root.logger=DEBUG,console and it started without any issues and there was no apparent errors in the log, only these two warnings but i dont think its relevant: WARNING: Use "yarn jar" to launch YARN applications. 16/05/23 10:41:17 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist after the service was started via the above command, I tried to "start" the service in ambari and it connected succesfully, also via the CLI with the following command: beeline -u 'jdbc:hive2://HOSTNAME:10000/;transportMode=binary' when I tried to stop the service from ambari, it showed me that the services was stopped but when I checked the server it was still running all of the above leads me to the conclusion that there is some problem between ambari and hiveserver2, its like its unable to start/stop the hiveserver2 service, and that is probably the reason why the log files are not update please advise on what could an issue between ambari and running the services? i dont have problems with other services running from ambari on the same server. Thanks. ... View more