Hi,   I have been requested to disable TLSv1 and TLSv1.1 from our Yarn service. Some background: we run a HDP cluster version 2.6.3.0-235 After scrapping the web for the specific configuration I need to disable the algorithms, the only solution I found is to configure "jdk.tls.disabledAlgorithms.   I have set it both in "java.security" file and in the jvm arguments themselves (via yarn-env setting in Ambari ).   In java.security:   jdk.tls.disabledAlgorithms=TLSv1, SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, SSL, SSLv2, TLSv1.1     When I check the running process I see the following jvm arguments (due to the setting in yarn-env):   /usr/jdk64/jdk1.8.0_112/bin/java -Dproc_resourcemanager -Xmx1024m -Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=zookeeper -Djava.security.auth.login.config=/etc/hadoop/2.6.3.0-235/0/yarn_jaas.conf -Dzookeeper.sasl.clientconfig=Client -Dhdp.version=2.6.3.0-235 -Djdk.tls.disabledAlgorithms=TLSv1,TLSv1.1 -Dhadoop...     But, when I check the supported TLS versions on the resource manager port (8190 in my case), TLSv1 and TLSv1.1 are still supported.   Any help, ideas, and suggestions on how to correctly configure the TLS version support would be appreciated. ... View more

I have installed Ambari with stand-alone database. I have invested a lot of time in setting the cluster (Spark dependencies for many projects, Yarn configurations, etc). I run ambari-server as root, HDP version is 2.6.3.0-235, OS is Ubuntu 16.04 I had experienced problems in stating the ambari-server (service ambari-server start), specifically, there were connection problems related to postgres. I found postgres-9.5 installed on that system and removed it (apt-get remove postgres-9.5). Now, all the problems have started, after I have done that, ambari-server service was no longer available. The problem I got: Failed to start ambari-server.service: Unit ambari-server.service not found. Now, I have tried running ambari-server install again and hope it would repair the problem (I saw ambari-server files at many locations on the machine but the service definition was missing). I have stumbled upon the problems: apt-get install ambari-server # resulted in the following error: Some packages could not be installed. This may mean that you haverequested an impossible situation or if you are using the unstabledistribution that some required packages have not yet been createdor been moved out of Incoming.The following information may help to resolve the situation:The following packages have unmet dependencies: ambari-server : Depends: postgresql (>= 8.1) but it is not going to be installedE: Unable to correct problems, you have held broken packages. apt-get upgrade ambari-server # resulted in the following error: Some packages could not be installed. This may mean that you haverequested an impossible situation or if you are using the unstabledistribution that some required packages have not yet been createdor been moved out of Incoming.The following information may help to resolve the situation:The following packages have unmet dependencies: postgresql : Depends: postgresql-11 but it is not going to be installedE: Broken packages I tried manually installing postgres 9.5 as before and I still get the error: Depends: postgresql (>= 8.1) I was not able to install postgres-11 due to missing dependencies (libicu52) not found. What can I do to fix it? ... View more

I have a Yarn+Spark cluster installed with Ambari (2.6.3.0-235). Upon this cluster I run Jupyter Notebook as a service. On a notebook, I connect to Spark using findSpark , I am getting a flood of error messages in the log: jupyter[1057]: [dispatcher-event-loop-1] INFO org.apache.hadoop.yarn.util.RackResolver - Resolved <machine-ip> to /default-rack I found this Jira task discussing exactly this problem, but I am not sure what is the easiest way to update this. Is there any log4j file I can just update to disable RackResolver messages to notebooks? ... View more

Hi everyone, have a cluster managed with Yarn and runs Spark jobs, the components were installed using Ambari (2.6.3.0-235). I have 6 hosts each with 6 cores. I use Fair scheduler I want Yarn to automatically add/remove executor cores, but no matter what I do it doesn't work Relevant Spark configuration (configured in Ambari): <code>spark.dynamicAllocation.schedu<wbr>lerBacklogTimeout 10s spark.dynamicAllocation.sustai<wbr>nedSchedulerBacklogTimeout 5s spark.driver.memory 4G spark.dynamicAllocation.enable<wbr>d true spark.dynamicAllocation.initia<wbr>lExecutors 6 (has no effect - starts with 2) spark.dynamicAllocation.maxExe<wbr>cutors 10 spark.dynamicAllocation.minExe<wbr>cutors 1 spark.scheduler.mode FAIR spark.shuffle.service.enabled true SPARK_EXECUTOR_MEMORY="36G" Relevant Yarn configuration (configured in Ambari): <code>yarn.nodemanager.aux-services mapreduce_shuffle,spark_shuffl<wbr>e,spark2_shuffle YARN Java heap size 4096 yarn.resourcemanager.scheduler<wbr>.class org.apache.hadoop.yarn.server.<wbr>resourcemanager.scheduler.fair<wbr>.FairScheduler yarn.scheduler.fair.preemption true yarn.nodemanager.aux-services.<wbr>spark2_shuffle.class org.apache.spark.network.yarn.<wbr>YarnShuffleService yarn.nodemanager.aux-services.<wbr>spark2_shuffle.classpath {{stack_root}}/${hdp.version}/<wbr>spark2/aux/* yarn.nodemanager.aux-services.<wbr>spark_shuffle.class org.apache.spark.network.yarn.<wbr>YarnShuffleService yarn.nodemanager.aux-services.<wbr>spark_shuffle.classpath {{stack_root}}/${hdp.version}/<wbr>spark/aux/* Minimum Container Size (VCores) 0 Maximum Container Size (VCores) 12 Number of virtual cores 12 Also I followed the manual in and passed all the steps to configure external shuffle service, I copied the yarn-shuffle jar: cp /usr/hdp/2.6.3.0-235/spark/aux/spark-2.2.0.2.6.3.0-235-yarn-shuffle.jar /usr/hdp/2.6.3.0-235/hadoop-yarn/lib/ I see only 3 cores are allocated to the application (default executors is 2 so I guess its 2+driver) screenshot from the queue is attached, although many tasks are pending (screenshot added). I want to get to a point where Yarn starts with 3 cpu for every application, but when there are pending tasks more resources are allocated. If it it relevant, I use Jupyter Notebook and findspark to connect to the cluster: import findspark findspark.init() spark = SparkSession.builder.appName("internal-external2").getOrCreate() I would really appreciate any suggestion/help, there is no manual on that topic I didn't try. thx a lot, Anton ... View more

Spark jobs are divided into stages which are divided into tasks. Where tasks are the unit which is executed on the executor with the driver orchestrating the computation order and the task per executor. Yarn allows resource management and has a preemption capability, which can task resources from one application to another. My question is which one of the units is actually preempted, suppose an executor's current calculation is canceled and it starts serving a different application which one of the units is canceled? If its task, a different executor may pick up the calculation (what about the intermediate results of the stage?) If its stage, the question is the same. And a job cancellation seems like too much. ... View more

I am new to Yarn but have some experience with Spark stand alone master, I have recently installed a Yarn+Spark cluster using Ambari. I have a spark program (program.jar) compiled to jar which relies on another jar to work (infra.jar). I set the following configurations in Ambari: spark.executor.extraClassPath=/root/infra.jar spark.driver.extraClassPath=/root/infra.jar I verified the file exists on all nodes and verified the configuration was pushed to $SPARK_HOME/conf/spark-defaults.conf on all nodes. Actually I have also copied it to $SPARK_HOME/jars on all nodes. I run the job using: $SPARK_HOME/bin/spark-submit --master yarn --class com.MyMainClass /root/program.jar I have the following env variables: export HADOOP_CONF_DIR=/usr/hdp/2.6.3.0-235/hadoop/conf export HADOOP_HOME=/usr/hdp/2.6.3.0-235/hadoop I am getting an error: Caused by: java.lang.ClassNotFoundException: com.MyPartition at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.JavaDeserializationStream$anon$1.resolveClass(JavaSerializer.scala:67) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:312) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) MyPartition class is located in infra.jar but for some reason it is not found. This code looks like it is called in the Executor , also some code does run - which is the driver code. P.S. I tried adding the jar manually either by --jars flag or by addJars it still fails with ClassNotFoundException(although for some weird reason on different class from infra.jar) ... View more