About arunpoy

arunpoy · ‎01-06-2017

We are running our ambari database with mysql on a dedicated host. Now we need to move the DB to a different host. what are the steps that needs to be done to make this migration smooth and retain all the data in the new database on the new host.

arunpoy · ‎01-05-2017

@Aravindan Vijayan, these 2 lines are not present in any of the 2 files mentiones in the url you gave. That means this is equivalent to getting commented out. so metrics at region level too are getting flooded. YEs the metrics collector host is cohosted with a datanode. but we are planning to move it to a dedicated admin host.

arunpoy · ‎01-04-2017

Hi @Aravindan Vijayan, I cleared of the ambari metrics data and restarted metrics again. but collector went down again with the following error. Guess it is due to lack of resources. can you please point out the parameter that needs to be increased for our cluster configuration. I have given the cluster details in the previous message. it is a 30 node cluster and we have 256 gb ram in 28 slave nodes. also i have attached the entire log after today's restart.ambari-metrics-collector.zip. 2017-01-04 02:22:52,065 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 4 actions to finish 2017-01-04 02:22:52,065 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 4 actions to finish 2017-01-04 02:22:52,065 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 4 actions to finish 2017-01-04 02:22:52,066 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 18 actions to finish 2017-01-04 02:22:52,067 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 1879 actions to finish 2017-01-04 02:22:53,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6 actions to finish 2017-01-04 02:22:53,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 74 actions to finish 2017-01-04 02:22:53,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 121 actions to finish 2017-01-04 02:22:53,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 74 actions to finish 2017-01-04 02:22:53,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6 actions to finish 2017-01-04 02:22:53,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 74 actions to finish 2017-01-04 02:22:53,880 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 43 actions to finish

arunpoy · ‎12-28-2016

Hi @Aravindan Vijayan, Sorry for the delayed reply. our cluster size is 30 nodes. I have attached the details you have asked for and this is the output of http:<AMS_HOST>:6188/ws/v1/timeline/metrics/metadata {"timestamp":0,"starttime":0,"metrics":{}}hbase-site.xmlams-site.xmlams-env.txtams-env.txthbase-env.txt Do we need to increase any of the parameters for metrics

arunpoy · ‎12-23-2016

@Rajkumar Singh, i removed the znode /hiveserver2 from zk after stopping hive service. restarted again.it is running fine for more than 20 mins. Let me see what happens. Thanks for your help

arunpoy · ‎12-23-2016

@Rajkumar Singh it is a quorum of 3 servers

arunpoy · ‎12-23-2016

hi @Rajkumar Singh, this is what i see in the ZK [zk: localhost(CONNECTED) 1] ls /hiveserver2 [serverUri=abcd.solutions.local:10000;version=1.2.1000.2.4.0.0-169;sequence=0000000000]

arunpoy · ‎12-23-2016

Hive server 2 goes down in our environment within 5 minutes of bringing it up with the following error, any thoughts 2016-12-22 13:17:31,526 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(236)) - Unable to create a znode for this server instance java.lang.Exception: Max znode creation wait time: 120s exhausted at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2016-12-22 13:17:31,531 INFO [main]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2 2016-12-22 13:17:31,531 INFO [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(199)) - Thrift server has stopped 2016-12-22 13:17:31,531 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped. 2016-12-22 13:17:31,532 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped. 2016-12-22 13:17:31,532 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped. 2016-12-22 13:17:41,533 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped. 2016-12-22 13:17:41,534 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped. 2016-12-22 13:17:41,544 INFO [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x358a9c9199506a2 closed 2016-12-22 13:17:41,545 INFO [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper. 2016-12-22 13:17:41,545 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(442)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds java.lang.Exception: Max znode creation wait time: 120s exhausted at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2016-12-22 13:17:41,546 INFO [main-EventThread]: zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down 2016-12-22 13:18:33,625 INFO [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down HiveServer2 at abc.solutions.local/172.16.3.196 ************************************************************/ 2016-12-22 13:18:33,631 INFO [Thread-7]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2 2016-12-22 13:18:33,632 INFO [Thread-7]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper

arunpoy · ‎12-22-2016

one quick question @Rajkumar Singh, can this property be set in the advnaced yarn-site section in ambari old <property> <name>yarn.resourcemanager.state-store.max-completed-applications</name> <value>${yarn.resourcemanager.max-completed-applications}</value> </property> new <property> <name>yarn.resourcemanager.state-store.max-completed-applications</name> <value>1000</value> </property>

arunpoy · ‎12-22-2016

Thanks @Rajkumar Singh, if i reduce this parameter and if i want to retrieve history will it be possible. say for example if i have set this value to 1000, will it be possible for me to retrieve jobs more than 1000 if needed?

Online	Offline
Last Visited	‎03-14-2019 05:48 AM

Member Since	‎01-21-2016 11:26 AM
Last Visited	‎03-14-2019 05:48 AM
Posts	290
Kudos received	76

Cloudera Community

Re: Dead region server showing on ambari even afte...

Re: Zeppelin phoenix error

Re: job history not available

Migrating ambari database from one host to other

Re: ambari metrics collector going down

Re: ambari metrics collector going down

Re: ambari metrics collector going down

Re: HiveServer 2 goes down

Re: HiveServer 2 goes down

Re: HiveServer 2 goes down

HiveServer 2 goes down

Re: yarn web ui is too slow

Re: yarn web ui is too slow