in our ambari cluster ( version 2.6 ) we have masters machines and workers machines while kafka installed on the master machines the partition /data is only 15G and kafka log folder is - /data/vars/kafka/kafka-logs most of the folders under /data/vars/kafka/kafka-logs are with size 4K-40K but two folders are very huge size - 5G-7G , and this cause /data to be 100% example: under /data/vars/kafka/kafka-logs/mmno.aso.prpl.proces-90 folder: -rw-r--r-- 1 kafka hadoop 1073037840 Oct 2 14:05 00000000000000000000.log -rw-r--r-- 1 kafka hadoop 9480 Oct 2 14:05 00000000000000000000.index -rw-r--r-- 1 kafka hadoop 13596 Oct 2 14:05 00000000000000000000.timeindex -rw-r--r-- 1 kafka hadoop 1073464387 Oct 2 14:45 00000000000001419960.log -rw-r--r-- 1 kafka hadoop 9632 Oct 2 14:45 00000000000001419960.index -rw-r--r-- 1 kafka hadoop 14412 Oct 2 14:45 00000000000001419960.timeindex -rw-r--r-- 1 kafka hadoop 1073132221 Oct 2 15:23 00000000000002840641.log du -sh * 12K 00000000000000000000.index 1.0G 00000000000000000000.log <--- is it possible to limit the size of the logs? or other solution ? ( some variables from the ambari GUI that need to add or to reconfigure ? ) kafka have small /data folder and logs should not be with 1G size , how to solve it? ... View more

how to change the NodeManager Java heap size value , this value cant change from the ambari GUI ( value is static ), how to enable in order to change it I need to change this parameter according to the solution - https://community.hortonworks.com/content/supportkb/49606/gc-overhead-limit-exceeded.html we have a problem to start the nodemanager on the worker machine example from the log on worker machine: FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager java.lang.OutOfMemoryError: GC overhead limit exceeded at org.fusesource.leveldbjni.internal.NativeIterator.key(NativeIterator.java:171) at org.fusesource.leveldbjni.internal.JniDBIterator.peekNext(JniDBIterator.java:87) at org.apache.hadoop.yarn.server.utils.LeveldbIterator.peekNext(LeveldbIterator.java:137) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainersState(NMLeveldbStateStoreService.java:169) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:262) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:246) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:546) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594) 2017-09-15 11:19:27,456 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException ... View more