Created 10-06-2017 10:42 AM
We have flume service in HDP 2.5 which has multiple agents in it. All agents are using JVM from flume-env.sh and spawning individual JVM. Few are using less memory and few more. Is it possible to customize the JVM properties for each agent?
Created 10-06-2017 11:43 AM
you can use the different flume-env.sh file for each agent and you can specify the parameters using:
export JAVA_OPTS="-Xms2048m -Xmx8196m -XX:MaxDirectMemorySize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC"
Hope this helps.
Created 10-06-2017 12:53 PM
But Flume service in Ambari has only one flume-env.sh!!!
Created 10-10-2017 12:39 PM
From Ambari you can only configure only one flume-env.sh file for the environment but this can achieved in two ways:
1) You can use Ambari config groups to have different flume-env.sh files for different hosts
2) If do not want to change the default config, you need to go with CLI option i.e. creating the different config files/folders for each flume agent and starting them from CLI only, but with this you will not be able to manage the flume agent's through Ambari (a limitation of Ambari)
Hope this helps.
Created 10-06-2017 07:25 PM
You could use Ambari Configuration groups to achieve that. See this thread.
Created 10-10-2017 06:49 AM
Edit the flume-env.sh under Ambari to add the parameter. Restart services, this should affect all the flume agents in the cluster.