About mike_bronson7

mike_bronson7 · ‎07-10-2018

about - Is also possible to configure rolling logs on yarn - so they have this option if they need to keep verbose but at least you could restrict the size of the logs and how many you like to keep since we are talking about the stdout file , we not want to roll this file , what we want is to limit the size for example max 1G

76_subhasis · ‎07-11-2018

HDP 2.6.4 to HDP 2.6.5 is a patch/maintenance HDP upgrade. I have recently done with with Ambari 2.6.2

jsensharma · ‎07-09-2018

@Michael Bronson Additionally please check if there is any issue in connecting to HiveSerevr2 using Beeline manually? This will give us some idea like how much time does the beeline connection takes. If it takes more than default hardcoded 30 secodns then please try to increase the timeout to a higher value like 60 in the same script: # grep 'timeout=30' /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py # grep 'timeout=30' /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py .

jsensharma · ‎06-16-2018

@Michael Bronson As the AMS collector seems to be running it should collect and show the data however we will need to check if AMS collector has the data or not? So in order to check that we will need to capture the following informations: 1. Need to make an API call to the AMS collector in order to list down the number & names of metrics that it is collecting? and from which hosts. The following URLs will give us JSON output . Can you please share the same here. http://<ams-host>:6188/ws/v1/timeline/metrics/metadata http://<ams-host>:6188/ws/v1/timeline/metrics/hosts 2. Need to check the Binaries of ambari collector and sink (so that we can see if it is same as ambari ersion) # rpm -qa | grep ambari 3. Can you please check if the metrics-monitor logs shows any kind of connectivity error while connecting to AMS collector? # /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out 4. Do we see if the hostname resolution if file and correctly defined inside the "/etc/ambari-metrics-monitor/conf/metric_monitor.ini" file 5. Please check if the AMS collector log shows any error?

mike_bronson7 · ‎06-12-2018

in the first zoo log we can see also the following: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,856 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)

mike_bronson7 · ‎06-28-2018

finally we success to decrease it by restart the postgresql and set the right values in postgresql.conf

Shelton · ‎06-10-2018

@Michael Bronson Sorry to get back late there is no real test. But you could retry the same manipulation by upload conf file to zookeper you shouldn't get the same error

Shelton · ‎06-10-2018

@Michael Bronson I meant using i.e Firewall to block the public-repo-1.hortonworks.com public IP or disable it in the repo by setting enabled=0 (false) instead of 1 (true) see below [HDP-UTILS-1.1.0.21] name=HDP-UTILS-1.1.0.21baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/xx path=/ enabled=0 That you can use a script

jsensharma · ‎06-07-2018

@Michael Bronson When we want to do GC log rotation then in that case we will have to define few properties together to make it work appropriately like following along with the "-Xloggc" option. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M Just defining NumberOfGClogFiles option will lead to inconsistent JVM options and JVM will not boot like: So please try this: HBASE_MASTER_OPTS=" -XX:PermSize=64m -XX:MaxPermSize={{hbase_master_maxperm_size}} -Xms{{hbase_heapsize}} -Xmx{{hbase_heapsize}} -Xmn{{hbase_master_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=20M" . NOTE: Also in your case the JVM was not booting because there is a Typo in your case "-XX:NumberOfGClogFiles" (here you used lowercase log) where as the correct option has Uppercase L in the option like "-XX:NumberOfGCLogFiles"

jsensharma · ‎06-07-2018

@Michael Bronson The Logs (like collector-gc.log Or gc.log) produced by the JVM following are not controlled by the Log4j So there is no way to control them or their rotations using Log4j -Xloggc:/var/log/ambari-metrics-collector/collector-gc.log-201806070708 -Xloggc:/var/log/ambari-metrics-collector/gc.log-201806070708 In order to control those GC log files you will need to rely on the OS utilities like "logrotate" or "cronjobs" to clear those logs. . JVM provides some options like "-XX:+UseGCLogFileRotation" however they are responsible for rotating the GC logs when the JVM process is keep running. However You can however control the AMS component logs like HMaster logs or AMS collector logs using the log4j settings like: log4j.appender.file.MaxFileSize={{ams_log_max_backup_size}}MB log4j.appender.file.MaxBackupIndex={{ams_log_number_of_backup_files}} .

Online	Offline
Last Visited	‎08-27-2024 09:17 AM

Member Since	‎08-08-2017 09:40 AM
Last Visited	‎08-27-2024 09:17 AM
Posts	1,652
Kudos received	29

Cloudera Community

Re: how to find number of CPU core on datanode ma...

Re: postgresql + ambari server failed to open port...

Re: how to stop the thrift servers by REST API

Re: namenode is in safe mode

Re: Directory /grid/sdg/hadoop/hdfs/data became un...

Re: yarn logs + stdout huge size

Re: is it possible to upgrade HDP from 2.6.4 to 2....

Re: hive + many hive process was open on namenode ...

Re: matrics collector is up but metrics dashboard ...

Re: zookeper - Too many connections

Re: How to clear /clean-up pg_xlog directory? on a...

Re: how to increase the ZNode size in ambari clus...

Re: how to disable ambari access to public-repo-1....

Re: Automatic deletion of rotated gc.log files

Re: how to limit the logs rotation under /var/log/...