Cloudera Community
mike_bronson7
user rank
Guru

Re: hive + many hive process was open on namenode ...

by Guru in Support Questions
‎07-09-2018 06:18 AM
‎07-09-2018 06:18 AM
not have the jstack command on my linux machine , should I download this cli from redhat repo? ... View more

hive + many hive process was open on namenode mach...

by Guru in Support Questions
‎07-08-2018 08:09 PM
‎07-08-2018 08:09 PM
we have ambari cluster version 2.6.0.x we noitce that free memory on master01 machine is 0 , and that because many of the following process ( from ps -ef | grep java ) ambari-+ 65369 65322 0 Jul06 ? 00:01:52 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.0.3-8 -Djava.net.preferIPv4Stack=true -Dhdvar/log/hadoop/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.0.3-8/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,c.6.0.3-8/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.0.3-8/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -.config.file=/usr/hdp/2.6.0.3-8/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullA/usr/hdp/2.6.0.3-8/hive/lib/hive-beeline-1.2.1000.2.6.0.3-8.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://master01.sys748.com:10000/;transportMo any idea why all this process ( around 350 process was opened ? ) , and take all memory on master01 machine [root@master01 ~]# ps -ef | grep java | wc -l 359 other way to show the process : ps -ef | sed 's/-D[^ ]*//g;s/-X[^ ]*//g;s#^.*/bin/java##g;s/[^ ]*.jar//g;s/^[ ]*//g' |more ambari-+ 50648 50646 0 Jul06 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/ *:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin /:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://master01.sys748.com:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' ... View more

Re: yarn + usercache + folder became with huge siz...

by Guru in Support Questions
‎07-04-2018 09:11 AM
‎07-04-2018 09:11 AM
@Jay I not have the variables in YARN - yarn.nodemanager.localizer.cache.target-size-mb , yarn.nodemanager.localizer.cache.cleanup.interval-ms: so please advice how to add them ? and what is the values that I need to set each of them ? note - /var is 100G on each worker machine ... View more

yarn + usercache + folder became with huge size

by Guru in Support Questions
‎07-04-2018 09:01 AM
‎07-04-2018 09:01 AM
hi all we have hadoop cluster version - 2.6.0.3 with yarn version - 2.7.3 we see that /var in workers ( data node ) machine is full and the root cause for this is that we see huge folders - lockmgr-b1ed0e9c-5700-4575-aa5e-182146f743d9 under /var/hadoop/yarn/local/usercache/hdfs/appcache/application_1530106922052_0041 please advice how to avoid this isshu , why folder are in that huge capasity ? [root@worker01 application_1530106922052_0041]# pwd /var/hadoop/yarn/local/usercache/hdfs/appcache/application_1530106922052_0041 [root@worker01 application_1530106922052_0041]# \ls -ltr total 8 drwx--x--- 2 yarn hadoop 6 Jul 3 06:19 filecache drwxr-xr-x 66 yarn hadoop 4096 Jul 3 06:24 blockmgr-b1ed0e9c-5700-4575-aa5e-182146f743d9 drwxr-xr-x 65 yarn hadoop 4096 Jul 4 08:02 blockmgr-c6530cea-1e98-419b-8653-3e9b467ac029 [root@worker01 application_1530106922052_0041]# du -sh * 33G blockmgr-b1ed0e9c-5700-4575-aa5e-182146f743d9 31G blockmgr-c6530cea-1e98-419b-8653-3e9b467ac029 ... View more
Labels:

Re: How to clear /clean-up pg_xlog directory? on a...

by Guru in Support Questions
‎06-28-2018 10:18 AM
‎06-28-2018 10:18 AM
finally we success to decrease it by restart the postgresql and set the right values in postgresql.conf ... View more

Re: How to clear /clean-up pg_xlog directory? on a...

by Guru in Support Questions
‎06-27-2018 09:58 AM
‎06-27-2018 09:58 AM
Hi Geoffrey , please advice which lines in postgresql.conf we need to comment ? ( archiving/replication ) ... View more

matrics collector is up but metrics dashboard not ...

by Guru in Support Questions
‎06-15-2018 02:51 PM
‎06-15-2018 02:51 PM
hi all we have ambari cluster , and matrics collector is up but metrics dashboard not display usage ( no data availble ) as the following ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:52 PM
‎06-12-2018 06:52 PM
in the first zoo log we can see also the following: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,856 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:50 PM
‎06-12-2018 06:50 PM
see also the following details ( sent much less then Received ) , is it can tell us something about the problem ? this happend only on the first zoo server echo stat | nc 10414.42.169 2181 Latency min/avg/max: 0/10/2727 Received: 600879 Sent: 103803 Connections: 30 Outstanding: 546 Zxid: 0x3e000048c3 Mode: follower Node count: 43296 ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:07 PM
‎06-12-2018 06:07 PM
I can kill all CLOSE-WAIT by - lsof -i :2181 | grep CLOSE_WAIT| awk '{print $2}' |uniq| xargs kill , but this isnt solution , why zoo not close them ? ... View more
Contact Me
Online
Offline
Last Visited
‎08-27-2024 09:17 AM
Community Statistics
Member Since ‎08-08-2017 09:40 AM
Last Visited ‎08-27-2024 09:17 AM
Posts 1,652
Kudos received 29