question Re: What services not needed? and can be removed in Support Questions

What services not needed? and can be removed

ilia987 — Wed, 20 Feb 2019 18:37:38 GMT

i have a large cluster installed currently it is used only for spark on yarn jobs

what services on the master i can remove\uninstall to free up some ram?

one server to accept spark jobs currently have the following services:

Timeline Service V1.5 / YARN Master
History Server / MapReduce2 Master
Hive Metastore / Hive Master
HiveServer2 / Hive Master
Metrics Collector / Ambari Metrics Master
Grafana / Ambari Metrics Master
NameNode / HDFS Master
ResourceManager / YARN Master
SNameNode / HDFS Master
Spark2 History Server / Spark2 Master
Timeline Service V2.0 Reader / YARN Master
YARN Registry DNS / YARN Master
ZooKeeper Server / ZooKeeper Master
DataNode / HDFS Slave
Metrics Monitor / Ambari Metrics Slave
HDFS Client / HDFS Client
Hive Client / Hive Client
MapReduce2 Client / MapReduce2 Client
Spark2 Client / Spark2 Client
Tez Client / Tez Client
YARN Client / YARN Client
ZooKeeper Client / ZooKeeper Client

and multiple clients with:

Metrics Monitor / Ambari Metrics Slave
NodeManager / YARN Slave

Re: What services not needed? and can be removed

jsensharma — Wed, 20 Feb 2019 18:54:32 GMT

@Ilia K

If you are not using Hive Service (HIve Metastore / HiveServer2 / Hive Client) then you can remove it.

Also AMS collector which internally starts an HBase instance to store the cluster & services related metrics data ... So if you are not much interested in Metrics Data then you can also Delete the AMS service (AMS Collector / Grafana / Metrics Monitors).

Re: What services not needed? and can be removed

ilia987 — Wed, 20 Feb 2019 19:02:50 GMT

Thanks for the quick response,

Can i delete all hive?

Hive Metastore
HiveServer2
Hive Client

Metrics are useful. i can see some bottlenecks and improve the performance,

Currently still struggling the improve the time it take to allocate the containers each job ( all the time they are the same and it take 30-60 seconds every job to allocate them )

Btw why it is a requirements (in the install stage it was selected cause i needed spark\yarn)

Re: What services not needed? and can be removed

jsensharma — Wed, 20 Feb 2019 19:09:29 GMT

@Ilia K

Bit correction here.

I just checkecd that Spark2 has a dependency on HIve Service.

# grep -A1 -B1 'HIVE' /var/lib/ambari-server/resources/common-services/SPARK2/2.0.0/metainfo.xml
            <dependency>
              <name>HIVE/HIVE_METASTORE</name>
              <scope>cluster</scope>

Hence you should not delete the HIVE service .. but if you are not using Hive then just Stop the "Hive Service" components that way it will not use RAM and put the Hive Service in Maintenance Mode.

Re: What services not needed? and can be removed

ilia987 — Wed, 20 Feb 2019 19:15:01 GMT

ok, just stopped 3 hive services, still it abit bother me that there are red dots in the ui :),

Btw why there are a dependency on hive?

Re: What services not needed? and can be removed

jsensharma — Wed, 20 Feb 2019 19:21:48 GMT

@Ilia K

Spark can be used to interact with Hive.

When you install Spark using Ambari, the hive-site.xml file is automatically populated with the Hive metastore location.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/spark-config-hive.html