About JLo_Hernandez

JLo_Hernandez · ‎09-13-2018

Hello team. I've just applied the recommended smartsense setting to make my dfs.namenode.checkpoint.period more efficent and I want to take a measure on how fast is now my namenode recovering from an restart than it was before. Where can I get an old lecture from the nn last restart and how much time did it take? Regards!

JLo_Hernandez · ‎09-13-2018

Below the answer in case someone need it: "Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

JLo_Hernandez · ‎09-12-2018

I am following Smartsense recommendation to make my cluster operationally more efficient, however there is a lot of documentation on how to set mentioned feature within ambari an why handlers are important for RPC, no one is mentioning why having this feature set "too high" affects a cluster. Can anyone share some thought here? Regards!"

JLo_Hernandez · ‎06-05-2018

It worked for me. Thanks!

JLo_Hernandez · ‎05-16-2018

Hey Geoffrey eventhough it worked. I kept monitoring it for a while and metrics went away again, but this time with a different message 2018-05-16 22:53:52,754 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Wed May 16 22:53:52 UTC 2018 2018-05-16 22:54:10,428 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:20,432 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:30,437 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:40,446 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:45,499 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Wed May 16 22:54:45 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Wed May 16 22:52:00 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Wed May 16 22:52:00 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Last check point time: 1526511120000, lagBy: 165 seconds. 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Wed May 16 22:54:45 UTC 2018, startTime = Wed May 16 22:52:00 UTC 2018, endTime = Wed May 16 22:54:00 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Skipping aggregation for metric patterns : sdisk\_%,boottime 2018-05-16 22:54:50,453 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:00,460 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:10,462 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:20,463 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:30,473 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:40,476 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:50,487 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:56:00,490 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:56:10,494 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish <br> and it's showing above all time. Any idea? I meant, the message is preatty clear and looks like my heap size is not enough for the amount of data the service is getting, this is what I have configure on my metrics collector heap size: metrics_collector_heapsize = 6144 If I have a cluster with 126 node and 106 of them has 899.50 GB as configure capaciy ant 20 of them with 399.75 GB what would be a fair amount of heap size to assign to this service does a formula exists for this? Regards!

JLo_Hernandez · ‎05-16-2018

Thank you Geoffrey it works and now logs are behaving as expected adn dashboard as well.

JLo_Hernandez · ‎05-16-2018

Problem: Amabri metrics dashboard within Ambari is not loading information (image attached) Actions: Ambari metrics service restart, Ambari collector restart. ambari-metrics-dashboard.png Also Ive founded this message on the logs. Therefore, I am trying to get mentioned server out from blacklist but I can't find information about it. 2018-05-16 20:32:17,258 [WARNING] emitter.py:146 - Error sending metrics to server. 2018-05-16 20:32:17,258 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:35:42,259 [WARNING] emitter.py:146 - Error sending metrics to server. '' 2018-05-16 20:35:42,260 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:35:47,260 [WARNING] emitter.py:120 - Metric collector host <server_name> was blacklisted. 2018-05-16 20:35:47,260 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:36:47,269 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:37:47,273 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:38:47,276 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:39:47,280 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:40:47,283 [INFO] emitter.py:154 - Calculated collector shard based on hostname : <server_name> 2018-05-16 20:44:07,284 [WARNING] emitter.py:146 - Error sending metrics to server. '' 2018-05-16 20:44:07,285 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:44:12,285 [WARNING] emitter.py:146 - Error sending metrics to server. 2018-05-16 20:44:12,285 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:47:37,286 [WARNING] emitter.py:146 - Error sending metrics to server. '' 2018-05-16 20:47:37,287 [WARNING] emitter.py:111 - Retrying after 5 ...

JLo_Hernandez · ‎04-27-2018

This is not an issue any more somebody ran hive on safe mode therfore beeline.properties files was created. Check below link. It gave me the root cause, but in order to fix specifically my issue, I needed either to delete the beeline.properties file created, remove below property or comment it out within beeline.properties files located on /home/<user_account>/.beeline/beeline.properties beeline.hiveconfvariables={} https://issues.apache.org/jira/browse/HIVE-16116

JLo_Hernandez · ‎04-24-2018

I have 2 edge nodes where I can do this, one is working properly and the other is not. I want to know what should I need to check in order to clear posted message when accessing to hive through beeline? This is the one is not working: beeline -n srvc_ima_platform -u 'jdbc:hive2://server:2181,server:2181,server:2181/ea_fin;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;?tez.queue.name=srvc_platform' --showHeader=false --outputformat=tsv2 --hiveconf hive.fetch.task.conversion=none Exception in thread "main" java.lang.NullPointerException at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:677) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:777) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:491) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:474) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) And this is he one working properly, both cli are exactly the same: beeline -n srvc_ima_platform -u 'jdbc:hive2://server:2181,server:2181,server:2181/ea_fin;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;?tez.queue.name=srvc_platform' --showHeader=false --outputformat=tsv2 --hiveconf hive.fetch.task.conversion=none Connecting to jdbc:hive2://server:2181,server:2181,server:2181/ea_fin;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;?tez.queue.name=srvc_platform Connected to: Apache Hive (version 1.2.1000.2.6.0.24-2) Driver: Hive JDBC (version 1.2.1000.2.6.0.3-8) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1000.2.6.0.3-8 by Apache Hive 0: jdbc:hive2://server:2> Regards!!

JLo_Hernandez · ‎03-24-2018

HADOOP_NFS3_OPTS="-Xmx{{nfsgateway_heapsize}}m -Dhadoop.security.logger=ERROR,DRFAS ${HADOOP_NFS3_OPTS}"

Online	Offline
Last Visited	‎09-22-2020 02:25 PM

Member Since	‎01-14-2016 10:55 PM
Last Visited	‎09-22-2020 02:25 PM
Posts	49
Kudos received	4

Cloudera Community

Re: How to make "hdfs dfs -du" command show 2 size...

Re: "hadoop fs -du" command giving incorrect value

Re: What's the problem on having dfs.namenode.hand...

Re: Getting "Exception in thread "main" java.lang....

Re: How to restart just spark client on Ambari

How to check how long did Namenode take to restart...

Re: What's the problem on having dfs.namenode.hand...

What's the problem on having dfs.namenode.handler....

Re: How to reset root password for sandbox hdp 2.4

Re: Ambari 2.6.* - Amabri Metrics UI is not loadin...

Re: Ambari 2.6.* - Amabri Metrics UI is not loadin...

Ambari 2.6.* - Amabri Metrics UI is not loading in...

Re: Getting "Exception in thread "main" java.lang....

Getting "Exception in thread "main" java.lang.Null...

How can I know specified JVM heap space (HADOOP_NF...