I have installed Ambari with stand-alone database. I have invested a lot of time in setting the cluster (Spark dependencies for many projects, Yarn configurations, etc). I run ambari-server as root, HDP version is 2.6.3.0-235, OS is Ubuntu 16.04 I had experienced problems in stating the ambari-server (service ambari-server start), specifically, there were connection problems related to postgres. I found postgres-9.5 installed on that system and removed it (apt-get remove postgres-9.5). Now, all the problems have started, after I have done that, ambari-server service was no longer available. The problem I got: Failed to start ambari-server.service: Unit ambari-server.service not found. Now, I have tried running ambari-server install again and hope it would repair the problem (I saw ambari-server files at many locations on the machine but the service definition was missing). I have stumbled upon the problems: apt-get install ambari-server # resulted in the following error: Some packages could not be installed. This may mean that you haverequested an impossible situation or if you are using the unstabledistribution that some required packages have not yet been createdor been moved out of Incoming.The following information may help to resolve the situation:The following packages have unmet dependencies: ambari-server : Depends: postgresql (>= 8.1) but it is not going to be installedE: Unable to correct problems, you have held broken packages. apt-get upgrade ambari-server # resulted in the following error: Some packages could not be installed. This may mean that you haverequested an impossible situation or if you are using the unstabledistribution that some required packages have not yet been createdor been moved out of Incoming.The following information may help to resolve the situation:The following packages have unmet dependencies: postgresql : Depends: postgresql-11 but it is not going to be installedE: Broken packages I tried manually installing postgres 9.5 as before and I still get the error: Depends: postgresql (>= 8.1) I was not able to install postgres-11 due to missing dependencies (libicu52) not found. What can I do to fix it? ... View more

Hi everyone, have a cluster managed with Yarn and runs Spark jobs, the components were installed using Ambari (2.6.3.0-235). I have 6 hosts each with 6 cores. I use Fair scheduler I want Yarn to automatically add/remove executor cores, but no matter what I do it doesn't work Relevant Spark configuration (configured in Ambari): <code>spark.dynamicAllocation.schedu<wbr>lerBacklogTimeout 10s spark.dynamicAllocation.sustai<wbr>nedSchedulerBacklogTimeout 5s spark.driver.memory 4G spark.dynamicAllocation.enable<wbr>d true spark.dynamicAllocation.initia<wbr>lExecutors 6 (has no effect - starts with 2) spark.dynamicAllocation.maxExe<wbr>cutors 10 spark.dynamicAllocation.minExe<wbr>cutors 1 spark.scheduler.mode FAIR spark.shuffle.service.enabled true SPARK_EXECUTOR_MEMORY="36G" Relevant Yarn configuration (configured in Ambari): <code>yarn.nodemanager.aux-services mapreduce_shuffle,spark_shuffl<wbr>e,spark2_shuffle YARN Java heap size 4096 yarn.resourcemanager.scheduler<wbr>.class org.apache.hadoop.yarn.server.<wbr>resourcemanager.scheduler.fair<wbr>.FairScheduler yarn.scheduler.fair.preemption true yarn.nodemanager.aux-services.<wbr>spark2_shuffle.class org.apache.spark.network.yarn.<wbr>YarnShuffleService yarn.nodemanager.aux-services.<wbr>spark2_shuffle.classpath {{stack_root}}/${hdp.version}/<wbr>spark2/aux/* yarn.nodemanager.aux-services.<wbr>spark_shuffle.class org.apache.spark.network.yarn.<wbr>YarnShuffleService yarn.nodemanager.aux-services.<wbr>spark_shuffle.classpath {{stack_root}}/${hdp.version}/<wbr>spark/aux/* Minimum Container Size (VCores) 0 Maximum Container Size (VCores) 12 Number of virtual cores 12 Also I followed the manual in and passed all the steps to configure external shuffle service, I copied the yarn-shuffle jar: cp /usr/hdp/2.6.3.0-235/spark/aux/spark-2.2.0.2.6.3.0-235-yarn-shuffle.jar /usr/hdp/2.6.3.0-235/hadoop-yarn/lib/ I see only 3 cores are allocated to the application (default executors is 2 so I guess its 2+driver) screenshot from the queue is attached, although many tasks are pending (screenshot added). I want to get to a point where Yarn starts with 3 cpu for every application, but when there are pending tasks more resources are allocated. If it it relevant, I use Jupyter Notebook and findspark to connect to the cluster: import findspark findspark.init() spark = SparkSession.builder.appName("internal-external2").getOrCreate() I would really appreciate any suggestion/help, there is no manual on that topic I didn't try. thx a lot, Anton ... View more