Member since
08-08-2017
1523
Posts
12
Kudos Received
10
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 196 | 01-30-2020 08:04 PM | |
| 137 | 07-07-2019 09:06 PM | |
| 703 | 01-27-2018 10:17 PM | |
| 521 | 12-31-2017 10:12 PM | |
| 342 | 12-03-2017 02:26 PM |
03-22-2020
12:38 AM
We have ambari cluster - HDP version 2.6.5
from time to time we see under replica in ambari dashboard , ( it means that under replica need to fix )
When I saw this, I ran the HDFS command that fixes it.
but I want to know if we can set some parameters that will do it automatically , by changing some values from ambari configuration.
... View more
Labels:
03-04-2020
12:23 PM
hi all
we have production HDP 2.6.4 cluster , with 12 data nodes machines
from ambari we can see ~25000 number of replica
when this happens then we fix the under replica
but after some time its return again
what we need to check in order to find the root cause for this behavior
... View more
Labels:
03-04-2020
02:01 AM
hi all
since we have from time to time under replica blocks
I want to know if we can do some cron-job to run the fix every 1 day
is it good idea to run the fix for under replica by cron job?
the procedure is
su - <$hdfs_user>
bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done
... View more
03-03-2020
01:38 AM
hi all
is it possible to get the number of mounted volumes on the first data-node machine by REST API
for example
this is the output from df -h on the first datanode ( we have 3 mounted disks on each datanode machine )
/dev/sdb 1.8T 1.1T 744G 60% /hdfs/sdb /dev/sdc 1.8T 1.1T 716G 61% /hdfs/sdc /dev/sdd 1.8T 1.1T 714G 62% /hdfs/sdd
how to get thehdfs mounted disks by ambari rest API?
... View more
Labels:
02-27-2020
04:25 PM
1 Kudo
Dear Jay when we run the API we get the following master02 - is the machine with ambari server kafka01 - is the first kafka machine where we are wrong ? curl -iskH "X-Requested-By: ambari" -u admin:admin -X PUT -d '{"RequestInfo":{"context":"Stop Kafka Broker","operation_level":{"level":"HOST_COMPONENT","cluster_name":"NwCluster","host_name":"kafka01","service_name":"KAFKA"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://master02:8080/api/v1/clusters/HDP_SYS76/hosts/kafka01/host_components/KAFKA_BROKER HTTP/1.1 404 Not Found X-Frame-Options: DENY X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Cache-Control: no-store Pragma: no-cache Set-Cookie: AMBARISESSIONID=15lniq8gcyggwudtsp5sd70fp;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT User: admin Content-Type: text/plain Content-Length: 243 { "status" : 404, "message" : "org.apache.ambari.server.controller.spi.NoSuchParentResourceException: Parent Host resource doesn't exist. Host not found, cluster=HDP_SYS76, hostname=kafka01. Host not found, cluster=HDP_SYS76, hostname=kafka01"
... View more
02-27-2020
04:11 PM
Dear Jay +1 for your excellent answer I will test this API soon when I will come to office , I will update soon
... View more
02-27-2020
01:39 PM
1 Kudo
hi all
in ambari we have kafka service , and this service include 7 kafka brokers
how to stop/start the kafka broker on specific host?
in this example kafka service contain the following kafka broker hosts
kafka01
kafka02
kafka03
kafka04
kafka05
kafka06
kafka07
other repleted post - https://community.cloudera.com/t5/Support-Questions/how-to-identify-alert-from-kafka-service-in-ambari-by-REST/m-p/284675#M211372
... View more
Labels:
02-27-2020
08:30 AM
hi all
in our cluster HDP - 2.6.4 , we have kafka cluster with 11 nodes ( 11 machines )
restart the kafka service , from AMBARI REST API , will stop/start is brutal
when we do it from rest API then its cause corrupted indexes files in kafka topic partitions
what we search is , ambari rest API that do the restart in parallel
for example
first stop kafka01
then
start kafka01
wait until alert gone (from ambari)
then stop kafka02 , and start kafka02
etc....
is this process could be done by ambari rest API?
... View more
Labels:
02-27-2020
06:28 AM
Hi all,
As all know in ambari metrics collector we can set embedded mode or distrusted mode
We try to set distrusted mode without success , since metrics collector fails after some time
So we back to embedded mode
Since embedded mode , actually wrote all data on local disk
We want to decrease the retention and minimize the metrics data on the local disk
What are the parameters in ambari metrics collector that help us to minimize the data on the local disk?
I see also this post , but not clear what the parameters that need to tune
https://community.cloudera.com/t5/Support-Questions/How-can-we-delete-old-data-of-Ambari-Metric-server/td-p/121736
... View more
Labels:
02-26-2020
08:19 AM
we have ambari with hadoop ( HDP 2.6.4 ) cluster
in ambari we have the ambari-metrics-collector
from time to time metrics collector is crashed and we can see from the log the following
note - this happens on both configuration - embedded and distributed
java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.helix.controller.stages.ClusterEventBlockingQueue.take(ClusterEventBlockingQueue.java:85) at org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:594) 2019-11-26 00:57:20,639 INFO org.apache.helix.controller.GenericHelixController: END ClusterEventProcessor thread 2019-11-26 00:57:20,641 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys67.com/130.21.6.36:61181. Will not attempt to authenticate using SASL (unknown error) 2019-11-26 00:57:20,642 WARN org.apache.zookeeper.ClientCnxn: Session 0x16df2739ea90013 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) 2019-11-26 00:57:20,686 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:6188 2019-11-26 00:57:20,690 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR javax.ws.rs.WebApplicationException: org.apache.phoenix.exception.PhoenixIOException: Interrupted calling coprocessor service org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for row \x00\x00METRIC_RECORD at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:377) at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1409) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) ... 48 more Caused by: java.io.InterruptedIOException: Giving up trying to location region in meta: thread is interrupted. at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1347) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1149) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1106) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:941) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
not clearly why we get
2019-11-26 00:57:20,690 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
I must to say that we tune the metrics collector parameters according to - https://cwiki.apache.org/confluence/display/AMBARI/Configurations+-+Tuning
so we not understand why ambari metrics collector failed
machines with ambari metrics collector include the following rpm's
ambari-metrics-monitor-2.6.2.2-1.x86_64 ambari-metrics-common-2.6.2.2-1.noarch ambari-agent-2.6.2.2-1.x86_64 ambari-metrics-collector-2.6.2.2-1.x86_64 ambari-metrics-hadoop-sink-2.6.2.2-1.x86_64 ambari-server-2.6.2.2-1.x86_64
... View more
02-25-2020
10:15 AM
another thing ls -ltr /var/lib/ambari-metrics-collector/hbase-tmp/ total 0 is empty ( since we remove the data from this folder for cleaning )
... View more
02-25-2020
10:12 AM
from - hbase-ams-master-master02 Tue Feb 25 18:07:05 UTC 2020, null, java.net.SocketTimeoutException: callTimeout=300000, callDuration=310495: row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=master02.sys67.com,61320,1582652433392, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89) at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:184) at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:136) at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:238) at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:118) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: callTimeout=300000, callDuration=310495: row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=master02.sys67.com,61320,1582652433392, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) ... 3 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:410) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:716) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1199) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:379) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:201) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ... 4 more 2020-02-25 18:07:05,670 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /ams-hbase-unsecure/splitWAL/RESCAN0000000390 entered state: DONE master02.sys67.com,61300,1582652432230 2020-02-25 18:07:06,545 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /ams-hbase-unsecure/splitWAL/RESCAN0000000391 entered state: DONE master02.sys67.com,61300,1582652432230
... View more
02-25-2020
10:07 AM
Dear Jay one of the logs tail -f ambari-metrics-collector.log ,612 INFO org.apache.helix.monitoring.mbeans.ClusterStatusMonitor: Unregistering ClusterStatus: cluster=ambari-metrics-cluster,instanceName=master02.sys67.com_12001,resourceName=METRIC_AGGREGATORS 2020-02-25 17:55:36,612 INFO org.apache.helix.monitoring.mbeans.ClusterStatusMonitor: Unregistering ClusterStatus: cluster=ambari-metrics-cluster 2020-02-25 17:55:36,612 INFO org.apache.helix.manager.zk.CallbackHandler: 117 END:INVOKE /ambari-metrics-cluster/CONTROLLER listener:org.apache.helix.manager.zk.DistributedLeaderElection Took: 4ms 2020-02-25 17:55:36,612 INFO org.apache.helix.manager.zk.ZkClient: Closing zkclient: State:CONNECTED Timeout:30000 sessionid:0x1707d6c38190018 local:/43.6.53.28:45161 remoteserver:master01.sys67.com/43.6.53.27:2181 lastZxid:12249246728803 xid:533 sent:537 recv:547 queuedpkts:1 pendingresp:0 queuedevents:0 2020-02-25 17:55:36,613 WARN org.apache.helix.manager.zk.CallbackHandler: Skip processing callbacks for listener: org.apache.helix.controller.GenericHelixController@61dde151, path: /ambari-metrics-cluster/LIVEINSTANCES, expected types: [INIT] but was CALLBACK 2020-02-25 17:55:36,629 INFO org.apache.helix.controller.GenericHelixController: Get FINALIZE notification, skip the pipeline. Event :idealStateChange 2020-02-25 17:55:36,643 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1707d6c38190018 closed 2020-02-25 17:55:36,643 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient 2020-02-25 17:55:36,643 INFO org.apache.helix.manager.zk.ZKHelixManager: Cluster manager: master02.sys67.com disconnected 2020-02-25 17:55:36,643 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x1707d6c38190018 2020-02-25 17:55:36,646 WARN org.apache.helix.controller.GenericHelixController: ClusterEventProcessor interrupted java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.helix.controller.stages.ClusterEventBlockingQueue.take(ClusterEventBlockingQueue.java:85) at org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:594) 2020-02-25 17:55:36,647 INFO org.apache.helix.controller.GenericHelixController: END ClusterEventProcessor thread 2020-02-25 17:55:36,671 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at master02.sys67.com/43.6.53.28 ************************************************************/
... View more
02-24-2020
10:11 PM
Dear Jay what to say - you are the best , well done
... View more
02-24-2020
09:45 PM
also this is what we get from "ops" in ambari you can see that ambari identify this as "$service" and not as KAFKA
... View more
02-24-2020
09:40 PM
Dear Jay yes I put KAFKA and not kafka , ( just I wrote by mistake in the post kafka instead KAFKA but with the same problem
... View more
02-24-2020
02:43 PM
the following API example , will stop the kafka service in ambari
export service = kafka
curl - u admin : admin - i - H 'X-Requested-By: ambari' - X PUT - d '{"RequestInfo":{"context":"_PARSE_.STOP.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http :// $HOST : 8080 / api / v1 / clusters / $CLUSTER_NAME / services / $service
the problem is about the syntax - _PARSE_.STOP.$service
and we see that actually service value - kafka not set in - PARSE.STOP.$service
so ambri saw the name as - PARSE.STOP.$service and not PARSE.STOP.kafka
any idea how we can set the value kafka inside the json syntax?
... View more
Labels:
02-23-2020
07:53 AM
hi all
metrics collector crash after moving to distributed mode and after metrics cleaning
metrics collector failed after some time and we get the following errors
2020-02-23 13:44:53,163 INFO org.apache.zookeeper.ZooKeeper: Session: 0x170721fadf90010 closed 2020-02-23 13:44:53,163 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient 2020-02-23 13:44:53,163 INFO org.apache.helix.manager.zk.ZKHelixManager: Cluster manager: master.sys9378.com disconnected 2020-02-23 13:44:53,163 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x170721fadf90010 2020-02-23 13:44:53,166 WARN org.apache.helix.controller.GenericHelixController: ClusterEventProcessor interrupted java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.helix.controller.stages.ClusterEventBlockingQueue.take(ClusterEventBlockingQueue.java:85) at org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:594) 2020-02-23 13:44:53,166 INFO org.apache.helix.controller.GenericHelixController: END ClusterEventProcessor thread 2020-02-23 13:44:53,173 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system... 2020-02-23 13:44:53,181 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped. 2020-02-23 13:44:53,181 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete. 2020-02-23 13:44:53,182 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory 2020-02-23 13:44:53,183 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200 2020-02-23 13:44:53,191 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2020-02-23 13:44:53,191 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200 2020-02-23 13:44:53,206 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at master.sys9378.com/42.3.44..126 ************************************************************/
not clearly why we get this errors ,because we clean all metrics data as
from temp folder
/hadoop/var/lib/ambari-metrics-collector
from HDFS
hdfs dfs -rm -r -f /user/ams/hbase/*
and from zoocli
[zk: 127.0.0.1:2181(CONNECTED) 3] rmr /ams-hbase-unsecure
... View more
Labels:
02-22-2020
11:13 PM
hi all
NameNode Java heap size , parameter is very important in HDFS
how to know the optimal value for "NameNode Java heap size" according to memory size
for example we have different machines with 32G and 64G and 128G
what should be the values of "NameNode Java heap size" , regarding to memory as above
... View more
Labels:
02-22-2020
10:30 PM
hi all
since some ambari clusters are with implemented ambari-metrics-collector in embedded mode.
we want to decrease the metrics data to 6 month , and purge the metrics that saved more then 6 month
is it possible?
... View more
Labels:
02-21-2020
12:23 AM
Dear Jay regarding this ambari cluster , I noticed that some of the metrics are shown and some of them are not as the following example from the logs we can see [root@master ambari-metrics-collector]# tail -10 hbase-ams-regionserver-master.sys77.com.log 2020-02-21 08:15:13,237 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://master.sys77.com:6188/ws/v1/timeline/metrics 2020-02-21 08:15:13,237 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://master.sys77.com:6188/ws/v1/timeline/containermetrics 2020-02-21 08:15:13,240 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 08:15:13,243 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 08:15:13,243 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 08:15:48,203 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Flush of region METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91. due to global heap pressure. Total Memstore size=227.6 M, Region memstore size=68.4 M 2020-02-21 08:15:48,205 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91., current region memstore size 68.37 MB, and 1/1 column families' memstores are being flushed. 2020-02-21 08:15:48,605 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3159, memsize=68.4 M, hasBloomFilter=false, into tmp file hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/.tmp/97bc3e982c0846e2864c2750526cca4f 2020-02-21 08:15:48,648 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/0/97bc3e982c0846e2864c2750526cca4f, entries=267808, sequenceid=3159, filesize=3.9 M 2020-02-21 08:15:48,652 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~68.37 MB/71693616, currentsize=0 B/0 for region METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91. in 447ms, sequenceid=3159, compaction requested=false [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# tail -10 ambari-metrics-collector.log 2020-02-21 08:16:20,918 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Fri Feb 21 08:16:20 UTC 2020 2020-02-21 08:16:21,227 INFO TimelineClusterAggregatorSecond: Saving 8394 metric aggregates. 2020-02-21 08:16:21,372 INFO TimelineMetricHostAggregatorMinute: 5448 row(s) updated in aggregation. 2020-02-21 08:16:21,419 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,440 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,457 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Fri Feb 21 08:10:00 UTC 2020, endTime = Fri Feb 21 08:15:00 UTC 2020 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Fri Feb 21 08:16:21 UTC 2020 2020-02-21 08:16:21,752 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Fri Feb 21 08:16:21 UTC 2020 [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# tail -10 hbase-ams-master-master.sys77.com.log 2020-02-21 07:45:19,565 INFO [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the METRIC_AGGREGATE_DAILY table succeeds 2020-02-21 07:46:09,290 INFO [timeline] availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard master.sys77.com based on hostname: master.sys77.com 2020-02-21 07:49:59,377 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2020-02-21 07:54:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2020-02-21 07:59:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0 2020-02-21 08:00:32,031 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hdfsha/user/ams/hbase/MasterProcWALs/state-00000000000000000055.log 2020-02-21 08:00:32,031 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hdfsha/user/ams/hbase/MasterProcWALs/state-00000000000000000056.log] 2020-02-21 08:04:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0 2020-02-21 08:09:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0 2020-02-21 08:14:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0 [root@master ambari-metrics-collector]# we can see that 2020-02-21 08:15:48,648 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/0/97bc3e982c0846e2864c2750526cca4f, entries=267808, sequenceid=3159, filesize=3.9 M but we set the - hbase.rootdir=/user/ams/hbase
... View more
02-20-2020
10:57 PM
Dear Jay yes change the ams-hbase-unsecure1 is part of the solution , also I increase the heap size since we saw also error about the heap size I will build more new cluster with your correction , and I will let you know if metrics working fine with hbase.rootdir=/user/ams/hbase
... View more
02-20-2020
07:39 PM
we configured the following this are the logs ( still metrics are now shown from ambari , and we have the same alert on the metrics service ) tail -15 ambari-metrics-collector.log at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167) 2020-02-21 03:33:49,284 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=15, retries=35, started=629725 ms ago, cancelled=false, msg=java.io.IOException: Table Namespace Manager not fully initialized, try again later at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2693) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2915) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1686) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:483) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:59846) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167) tail -30 hbase-ams-regionserver-master.sys77.com.log 2020-02-21 03:23:23,912 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2020-02-21 03:23:23,914 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 03:23:23,916 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 03:23:23,916 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 03:24:33,928 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-21 03:28:12,357 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=158.59 KB, freeSize=149.62 MB, max=149.78 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2020-02-21 03:28:13,923 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-21 03:28:23,344 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system... 2020-02-21 03:28:23,347 INFO [timeline] impl.MetricsSinkAdapter: timeline thread interrupted. 2020-02-21 03:28:23,349 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped. 2020-02-21 03:28:23,855 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2020-02-21 03:28:23,906 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2020-02-21 03:28:23,906 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = master.sys77.com, serviceName = ams-hbase 2020-02-21 03:28:23,909 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2020-02-21 03:28:23,911 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 03:28:23,914 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 03:28:23,914 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 03:29:33,925 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-21 03:33:12,356 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=158.59 KB, freeSize=149.62 MB, max=149.78 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2020-02-21 03:33:23,344 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system... 2020-02-21 03:33:23,346 INFO [timeline] impl.MetricsSinkAdapter: timeline thread interrupted. 2020-02-21 03:33:23,350 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped. 2020-02-21 03:33:23,853 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2020-02-21 03:33:23,924 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2020-02-21 03:33:23,925 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = master.sys77.com, serviceName = ams-hbase 2020-02-21 03:33:23,929 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2020-02-21 03:33:23,932 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 03:33:23,936 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 03:33:23,936 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 03:34:33,948 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. tail -30 hbase-ams-regionserver-master.sys77.com.log 2020-02-21 03:23:23,912 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2020-02-21 03:23:23,914 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 03:23:23,916 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 03:23:23,916 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 03:24:33,928 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-21 03:28:12,357 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=158.59 KB, freeSize=149.62 MB, max=149.78 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2020-02-21 03:28:13,923 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-21 03:28:23,344 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system... 2020-02-21 03:28:23,347 INFO [timeline] impl.MetricsSinkAdapter: timeline thread interrupted. 2020-02-21 03:28:23,349 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped. 2020-02-21 03:28:23,855 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2020-02-21 03:28:23,906 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2020-02-21 03:28:23,906 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = master.sys77.com, serviceName = ams-hbase 2020-02-21 03:28:23,909 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2020-02-21 03:28:23,911 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 03:28:23,914 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 03:28:23,914 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 03:29:33,925 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-21 03:33:12,356 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=158.59 KB, freeSize=149.62 MB, max=149.78 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2020-02-21 03:33:23,344 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system... 2020-02-21 03:33:23,346 INFO [timeline] impl.MetricsSinkAdapter: timeline thread interrupted. 2020-02-21 03:33:23,350 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped. 2020-02-21 03:33:23,853 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2020-02-21 03:33:23,924 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2020-02-21 03:33:23,925 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = master.sys77.com, serviceName = ams-hbase 2020-02-21 03:33:23,929 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2020-02-21 03:33:23,932 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 03:33:23,936 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 03:33:23,936 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 03:34:33,948 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. [root@master ambari-metrics-collector]#
... View more
02-20-2020
06:53 PM
Dear jay do you mean to set it as or
... View more
02-20-2020
12:18 PM
hi all
we have HDP version 2.6.5 and ambari 2.6.2.2
we configured the following
Metrics Service operation mode=distributed
hbase.rootdir=hdfs://hdfsha:8020/user/ams/hbase
and all the rest according to - https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode
so ambari configured to save the data on hdfs instead the local disk
after metrics collector service restart we saw the following from log
2020-02-20 20:09:15,770 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-02-20 20:11:35,769 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.
and
2020-02-20 20:30:10,306 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system started 2020-02-20 20:30:10,450 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2020-02-20 20:30:15,327 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
and without metrics as
and after some time the metrics collector service failed
why this happens ?
NOTE - I must to say that - when data of metrics saved on local disk then everything was ok
is that because reaching data on HDFS is more slow?
... View more
02-12-2020
10:24 PM
hi
we have ambari with HDP version 2.6.5
we want to clean all metrics data , according to the following instructions on link - https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data
so we did the following
note - Metrics Service operation mode - distributed
we stop the metrics service from ambari
we clean all data: ( from hdfs )
hdfs dfs -rm -r -f /apps/ams/metrics/*
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/.tmp' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/.tmp
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/MasterProcWALs' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/MasterProcWALs
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/WALs' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/WALs
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/archive' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/archive
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/data' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/data
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/hbase.id' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/hbase.id
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/hbase.version' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/hbase.version
20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/oldWALs' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/oldWALs
And we clean also the following folders
ls /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/
ls /var/lib/ambari-metrics-collector/hbase-tmp/phoenix-spool/
We start the metrics services from ambari
But from ambari metrics graphs not appears , and metrics collector service have alert
Not clearly why metrics not created after full metrics cleaning ,
From the log we can see the following:
2020-02-13 06:15:33,024 INFO [ProcedureExecutorThread-5] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=6 owner=ams state=ROLLEDBACK exec-time=239msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG
2020-02-13 06:15:44,356 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.
2020-02-13 06:16:21,223 INFO [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the SYSTEM.CATALOG table fails
2020-02-13 06:16:21,236 INFO [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null create 'SYSTEM.CATALOG', {TABLE_ATTRIBUTES => {PRIORITY => '2000', coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|805306366|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|805306367|'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
2020-02-13 06:16:21,349 INFO [ProcedureExecutorThread-6] procedure.CreateTableProcedure: CreateTableProcedure (table=SYSTEM.CATALOG) id=7 owner=ams state=RUNNABLE execute state=CREATE_TABLE_PRE_OPERATION
2020-02-13 06:16:21,360 WARN [ProcedureExecutorThread-6] procedure.CreateTableProcedure: The table SYSTEM.CATALOG does not exist in meta but has a znode. run hbck to fix inconsistencies.
2020-02-13 06:16:21,652 INFO [ProcedureExecutorThread-6] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=7 owner=ams state=ROLLEDBACK exec-time=305msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG
2020-02-13 06:17:14,354 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.
2020-02-13 06:17:58,076 INFO [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the SYSTEM.CATALOG table fails
2020-02-13 06:17:58,093 INFO [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null create 'SYSTEM.CATALOG', {TABLE_ATTRIBUTES => {PRIORITY => '2000', coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|805306366|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|805306367|'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
2020-02-13 06:17:58,206 INFO [ProcedureExecutorThread-7] procedure.CreateTableProcedure: CreateTableProcedure (table=SYSTEM.CATALOG) id=8 owner=ams state=RUNNABLE execute state=CREATE_TABLE_PRE_OPERATION
2020-02-13 06:17:58,218 WARN [ProcedureExecutorThread-7] procedure.CreateTableProcedure: The table SYSTEM.CATALOG does not exist in meta but has a znode. run hbck to fix inconsistencies.
2020-02-13 06:17:58,484 INFO [ProcedureExecutorThread-7] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=8 owner=ams state=ROLLEDBACK exec-time=279msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG
2020-02-13 06:19:24,358 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.
2020-02-13 06:19:34,540 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=156.56 KB, freeSize=147.69 MB, max=147.84 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=30, evicted=0, evictedPerRun=0.0
... View more
02-12-2020
04:41 AM
hi all
we have HDP version - 2.6.4
with active and standby namenode
is it possible to capture the active namenode IP by REST API?
we try by the following API , but API not return any output
curl -sH "X-Requested-By: ambari" -u admin:admin -H "X-Requested-By: ambari" -X GET http://master:8080/api/v1/clusters//services/HDFS
... View more
02-09-2020
06:29 PM
do you know , if some where we can found calculator for this parameter ? I mean lets say we have diff clusters and diff application running , and we want to tune this value according to our needs , then maybe some script that allow to set this value to the right tune?
... View more
02-09-2020
10:59 AM
we are running spark streaming app that consumes data from kafka topics
we want to know how to tune the parameter - `yarn.scheduler.capacity.maximum-am-resource-percent`
according to documentation:
yarn.scheduler.capacity.maximum-am-resource-percent: Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running,
on some document we even see that recomneded to utilise it to `90 percent` for best results,
but the default is `10%`
so my question is
do we need to tune this parameter according to cluster size?
or what is the best practice in order to get good results?
... View more
Labels:
02-09-2020
10:46 AM
we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform
we want to know if we can use the following HDFS components on docker containers as:
1. journal nodes
2. ZK fail controller
3. NameNode ( HA with active namenode and standby namenode )
after some searching , we not found so much documentation about HDFS on docker containers
so I am assume that HDFS on docker containers isnt good implementation
I will happy to get more ideas about this?
... View more
