How to calculate and change JVM Heap memory settings for Apache Knox service

[#1]

Unlike other services, Knox doesn't expose Java Heap settings via Ambari. Follow these steps to change default Heap settings for Knox:

1. On the Knox node, login as root and go to this directory: /usr/hdp/current/knox-server/bin

# cd /usr/hdp/current/knox-server/bin 

2. Make a copy of the file that we are going to change next.

# cp gateway.sh gateway.sh.backup.`date +%m%d%Y-%H-%M-%S`

3. Open gateway.sh in text editor and change this line:

APP_MEM_OPTS=""

to this:

APP_MEM_OPTS="-Xmx5g -XX:NewSize=3G -XX:MaxNewSize=3G -verbose:gc -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Xloggc:/var/log/knox/knox-gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"

Shown above are the example values for Knox Heap. This should be updated based on user's environment.

General formula for Knox Heap size will be:

Recommended KNOX Heap Size = 
(Webhdfs(ConcNumOfRequest*(replayBufferSize)) + 
HBase(ConcNumOfRequest*(replayBufferSize)) + 
Hive(ConcNumOfRequest*(replayBufferSize))) + 20 % for request surges

where:

ConcNumOfRequest = Number of concurrent requests expected for each component (WebHDFS, HBase, Hive etc.) replayBufferSize = Size of the largest incoming request to Knox

Based on the Heap size value, user need to further tune the NewSize and MaxNewSize.

4. Save changes to gateway.sh and restart Knox via Ambari. Confirm the new settings via 'ps -ef | grep knox' command.

# ps -ef| grep gateway
knox     29236     1 93 23:18 ?        00:00:29 /usr/jdk64/jdk1.8.0_77/bin/java -Xmx5g -XX:NewSize=3G -XX:MaxNewSize=3G -verbose:gc -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Xloggc:/var/log/knox/knox-gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -jar /usr/hdp/current/knox-server/bin/gateway.jar