question spark-shell --master yarn error in HDP 2.6.5 in Support Questions

question spark-shell --master yarn error in HDP 2.6.5 in Support Questions https://community.cloudera.com/t5/Support-Questions/spark-shell-master-yarn-error-in-HDP-2-6-5/m-p/280787#M208965 In HDP 2.6.5 I tried         <LI-CODE lang="markup">[root@sandbox-hdp ~]# spark-shell --master yarn</LI-CODE>       But got following errors. It opens scala shell but cannot create spark session.       <LI-CODE lang="markup">Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 19/10/19 06:04:17 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FAILED! 19/10/19 06:04:17 ERROR TransportClient: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException java.nio.channels.ClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) 19/10/19 06:04:17 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful java.io.IOException: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException .... .... Caused by: java.nio.channels.ClosedChannelException ... 19/10/19 06:04:17 ERROR Utils: Uncaught exception in thread Yarn application state monitor org.apache.spark.SparkException: Exception thrown in awaitResult: ... Caused by: java.io.IOException: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException ... 19/10/19 06:04:17 ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalStateException: Spark context stopped while waiting for backend ... console>:14: error: not found: value spark import spark.implicits._ ^ <console>:14: error: not found: value spark import spark.sql </LI-CODE>     Then I looked at yarn logs:   <LI-CODE lang="markup">19/10/19 06:21:35 INFO RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.2:8030 19/10/19 06:21:35 INFO YarnRMClient: Registering the ApplicationMaster 19/10/19 06:21:35 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead) .... 19/10/19 06:21:35 INFO RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.2:8030 19/10/19 06:21:35 INFO YarnRMClient: Registering the ApplicationMaster 19/10/19 06:21:35 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)</LI-CODE>     Sat, 19 Oct 2019 12:14:28 GMT erkansirin78 2019-10-19T12:14:28Z