CM interface to see exact settings

Question

Hi,

As there are what seems to be 1000s of settings for HDFS, Hive, CM, Hue, Spark, etc. I was wondering if there is a way to see the actual settings that are being used for all applications in the CM/Hadoop suite?

I know you can see some of them in config files, some of them using CM's interface. But not all of them are in both.

Defaults, for example, don't show up anywhere as sometimes code sets them. CM does not cover all the parameters as it has a special location it stores Safety Valve settings, as I've been told???

I would like to be able to see what is resident in memory and actually being used as you may add a setting and it not take as you could have put it in the wrong area or is being overridden by the same setting somewhere else.

Is there a way to actually see what is being used for any/all parameters in the suite?

Thanks!

benhadoop · Accepted Answer

Hi @pollard,looking at Impala Configs, you can find the /varz servlet at the debug WebUI of Impala Statestore, Catalogserver and Daemon. When you use the default ports these should be:http://state-store-host:25010/varzhttp://catalog-server-host:25020/varzhttp://impala-daemon-host:25000/varzOn the Impala Daemon, you also have a servlet for Hadoop vars:http://impala-daemon-host:25000/hadoop-varzBesides these servlets, Impala also prints it flags (which you were asking for in the second paragraph) during startup in the INFO-logs of each service. This may help, if your debug WebUIs are disabled for security reasons.For instance for Impala daemon: /var/log/impalad/impalad.INFOI0819 17:43:55.279785 18999 logging.cc:156] Flags (see also /varz are on debug webserver):--catalog_service_port=26000--catalog_topic_mode=full[…]--symbolize_stacktrace=false--v=1--vmodule= For other services, the /conf servlets at WebUIs or the Cloudera Manager configs (see other reply) mostly apply. If this or the first answer was helpful to you, please set it as accepted solution.Regards, Benjamin

benhadoop · Answer

Hi pollard,most services expose a "/conf" servlet in their WebUI which gives you the most complete set of actually used parameters. This should be the most promising source of thruth. Impala has a similar Servlet with path "/varz". The instance process view of Cloudera Manager shows you the actually distributed config files - which often helps a lot but does not include default values. You can reach it from a service (e.g. Impala) by clicking on "Instances" -> (the instance you want to see, e.g. Impala Catalog Server on a node) -> Processes Regards,Benjamin

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.