About benhadoop

benhadoop · ‎08-20-2019

Hi @R_SHETH, great to hear! Please mark the reply as the accepted answer if it solved your problem 🙂

benhadoop · ‎08-20-2019

Based on your output (auto mode in alternatives) you can try to add it with a higher priority (the 2.4 spark-submit has priority 10): /usr/sbin/alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 100 Be aware that Cloudera Manager might (try to) overwrite this on CDH updates. Again, I would recommend using the packaged Spark that comes with CDH and have doubts that using CDS in CDH 6.x is supported (also see [CDS Requirements - CDH Versions] [Migrating Apache Spark Before Upgrading to CDH 6]) Regards Benjamin

benhadoop · ‎08-20-2019

You can override the "spark-submit" association with alternatives, e.g.: /usr/sbin/alternatives --set spark-submit <path-to-spark2-submit> # <path-to-spark2-submit> could be like "/opt/cloudera/parcels/SPARK2-2.2.0-cloudera1-cdh5.13.3.p0.611179/bin/spark2-submit" # This is a path of CDH 5 to the Spark parcel directory. You need to adjust it to your path (the one that spark2-shell is pointing to) # You can find this path with: /usr/sbin/alternatives --display spark2-submit However, from CDH 6.x, it is normal to use spark-submit instead of spark2-submit because there is only Spark2 included in CDH. Also, it is normal to use the packaged version inside of the CDH distribution, which seems to be 2.4.0 for your CDH 6.1.1. I am not sure, if using a different version is supported or recommended by Cloudera. How did you install that Spark 2.2 version in your CDH 6 cluster? Also see https://www.cloudera.com/documentation/enterprise/6/6.1/topics/spark.html

benhadoop · ‎08-20-2019

Hi @pollard, looking at Impala Configs, you can find the /varz servlet at the debug WebUI of Impala Statestore, Catalogserver and Daemon. When you use the default ports these should be: http://state-store-host:25010/varz http://catalog-server-host:25020/varz http://impala-daemon-host:25000/varz On the Impala Daemon, you also have a servlet for Hadoop vars: http://impala-daemon-host:25000/hadoop-varz Besides these servlets, Impala also prints it flags (which you were asking for in the second paragraph) during startup in the INFO-logs of each service. This may help, if your debug WebUIs are disabled for security reasons. For instance for Impala daemon: /var/log/impalad/impalad.INFO I0819 17:43:55.279785 18999 logging.cc:156] Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --catalog_topic_mode=full […] --symbolize_stacktrace=false --v=1 --vmodule= For other services, the /conf servlets at WebUIs or the Cloudera Manager configs (see other reply) mostly apply. If this or the first answer was helpful to you, please set it as accepted solution. Regards, Benjamin

benhadoop · ‎08-19-2019

Hi pollard, most services expose a "/conf" servlet in their WebUI which gives you the most complete set of actually used parameters. This should be the most promising source of thruth. Impala has a similar Servlet with path "/varz". The instance process view of Cloudera Manager shows you the actually distributed config files - which often helps a lot but does not include default values. You can reach it from a service (e.g. Impala) by clicking on "Instances" -> (the instance you want to see, e.g. Impala Catalog Server on a node) -> Processes Regards, Benjamin

benhadoop · ‎08-15-2019

I am using CDH/CM 6.2. Will update the cluster and test again. However, according to the docs, it should already work since 5.14.

benhadoop · ‎08-15-2019

Hi community, looking at security, I am in process of disabling any interfaces without proper authentication / authorization (or even encryption). I came across the debug web UIs of Cloudera Management services. According to https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports_cm.html, the debug WebUIs can be disabled by setting the port property to -1. This works for Reports Manager (8083), Event Server (8084), Navigator Audit Server (8089), Telemetry Publisher (10111). This does not work, however, for Service Monitor (8086 / 9086 TLS), Activity Monitor (8087 / 9087 TLS), Host Monitor (8091 / 9091 TLS). Setting port to -1 leads to non-starting services without a proper ERROR in the log file. Cloudera Manager agent even tries to check, if the server successfully bound to port -1 and runs into errors: [15/Aug/2019 12:06:03 +0000] 65646 Thread-14 process ERROR [918-cloudera-mgmt-HOSTMONITOR] Failed port check: Command '['ss', '-np', 'state', 'listening', '(', 'sport', '=', '-1', 'or', 'sport', '=', '9995', 'or', 'sport', '=', '9994', ')']' returned non-zero exit status 255 How do you disable the debug web UIs for those management services. Or is there a way to properly secure them by authentication and authorization? Thanks and best regards Benjamin

Online	Offline
Last Visited	‎09-02-2019 06:07 AM

Member Since	‎12-14-2015 11:46 PM
Last Visited	‎09-02-2019 06:07 AM
Posts	89
Kudos received	7

Cloudera Community

Re: spark-submit still pointing to Spark-version a...

Re: CM interface to see exact settings

Re: spark-submit still pointing to Spark-version a...

Re: spark-submit still pointing to Spark-version a...

Re: spark-submit still pointing to Spark-version a...

Re: CM interface to see exact settings

Re: CM interface to see exact settings

Re: Disable Cloudera Management Debug WebUIs (Host...

Disable Cloudera Management Debug WebUIs (Host Mon...